Description
I've noticed a problem with unbounded memory usage in the consumer when handling lz4-compressed payloads. The symptoms point to a memory leak of some description. I've created a test case which should allow easy reproduction of the issue.
I've tried experimenting with various consumer settings such as fetch_max_bytes
, max_partition_fetch_bytes
and max_in_flight_requests_per_connection
in an attempt to lessen the buffering requirements of each individual request to Kafka. In all cases, memory usage of consumer processes has continued to rise until such a point that those processes are killed by the host system.
I can additionally confirm that this issue is present in the latest PyPI release (1.3.2) as well as master
and that the issue only manifests when using lz4
compression specifically. gzip
appears to be working fine, for example.
Any help is greatly appreciated. Thanks!