Don't encode negative ints as bcf_*_vector_end or other reserved values

[jmarshall]

The VCF v4.3 spec reserved a bunch of negative integers back in 2014 to allow for future expansion of missing / end-of-vector / etc special values:

Integers may be encoded as 8, 16, or 32 bit values, in little-endian order. It is up to the encoder to determine the appropriate ranged value to use when writing the BCF2 file. […] In total, eight values are reserved for future use: 0x80-0x87, 0x8000-0x8007, 0x80000000-0x80000007.

but it appears that this was never implemented in HTSlib.

In addition the previous code was avoiding bcf_int8/16_missing correctly but was wrongly encoding -127 and -32767 as bcf_int8/16_vector_end.

This patch is sufficient to fix samtools/bcftools#874 but it's possible that other code also needs modifying to properly reserve these values.

Also adds a test case exercising -127 and -128.

[pd3]

Thanks for looking at this. I think the same should be done in bcf_enc_vint(). Currently it works because the condition checks for the vector_end rather than missing, but the 8 reserved values are not respected.

[jmarshall]

You are quite right. I thought I looked at that function too, but apparently not carefully enough!
I'll amend the PR.

[jmarshall]

bcf_enc_vint() also now fixed.

This is probably NEWS-worthy, as previously bcftools has been writing slightly incorrect .bcf files. (Previously-written files remain readable with bcftools after this bug fix.)

[pd3]

Thank you

[daviesrob]

Thanks, merged with NEWS update.

[jmarshall]

Thanks. Um, I think it's -121…-128 and -32761…-32768 that get bumped up to the next length.

Re the second NEWS paragraph: BTW I haven't investigated fully when it prints out -127 in non-vector context as hoped for and when (like in the filter expression that exposed this) it sees it as vector_end. But it looks like plain converting does indeed work. Usually, hopefully 😄

[daviesrob]

Yes, you're right. 0x88 and 0x8008 escape unchanged. I'll modify the text.

I checked what happens in both scalar and vector contexts. It does appear to work in both as far as writing out VCF. Probably not safe to claim anything more than that though...