The serialization layer is unexpectedly processed before the producer's partitioning logic

[jeffwidman]

sample code pulled from one of our internal applications:

# kafka_producer is configured with:
#    "key_serializer": json.dumps,
#    "value_serializer": json.dumps,

key = None  # None produces round-robin
if Const.FIELD_USER in message:
    key = message[Const.FIELD_USER]
kafka_producer.send(topic, key=key, value=message)

Unsurprisingly, using json.dumps will serialize key=None to 'null'.

Surprisingly, this results in key=None behaving as if it were a keyed message and always being sent to a single partition rather than round-robining.

This is because the serialization layer is processed before the partitioning logic. So by the time https://github.com/dpkp/kafka-python/blob/1.4.4/kafka/partitioner/default.py#L24 is hit, the key is already the string 'null'.

I found this extremely surprising... at a minimum we need to call this out in the docs.

Alternatively, we could offer default helpers that handle null keys/values (for deleting messages in compacted topics) in a less surprising way.

Related: #913.

[tvoinarovskyi]

Hmm, yes, totally agree, at least a set of helpers would be good. I still find it kind of strange that None is even passed to the serializer, not just left as None. But we can't just change that at this point.

StringSerializer and JsonSerializer would at least be great. There is still an open question about what do we do with headers. Java has calls to serialize those too.