Misdetection of xerial snappy on RedPanda

[vmarkovtsev]

We consume messages from a Kafka topic from RedPanda that are compressed with Snappy. Apparently, RedPanda is doing something not completely standard or with a newer format version, since kafka.codec._detect_xerial_stream fails to detect xerial, returns False, and the snappy_decode crashes with snappy.UncompressError: Error while decompressing: invalid input. Here are the first few bytes of a sample payload:

\x82SNAPPY\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\xc3\x9b\x80\x80\x08\xf0\xaa\x98\x05\x00\x00

confluence-kafka works with RedPanda without a problem.

[dpkp]

It looks to me like the version and compat fields are (incorrectly) encoded in little-endian (1 => \x01\x00\x00\x00\). I wonder if confluence driver is not checking the version / compat fields ?

[dpkp]

Yea, looks like librdkafka only checks the SNAPPY magic string and ignores the version + compatibility fields. That's technically wrong, but it does avoid crashing on the strictly incorrect encoding from redpanda. https://github.com/confluentinc/librdkafka/blob/a09ccd9ffb3e4711dedf99aceb92e7056a7a9724/src/rdkafka_msgset_reader.c#L324

[dpkp]

Dropped version + compat validation in #2483