Fatal Python error: deallocating None when consuming lz4-compressed topic

[kiri11]

Python 3.5.1, kafka 0.10.1.0, kafka-python 1.3.3.
Problem arises when consuming a lot of messages (more than 15000 in the queue).
Can this be caused by #1024?

In kafka-python 1.3.2 there was no such error, but huge memory consumption.

Fatal Python error: deallocating None

Current thread 0x00007f7411669700 (most recent call first):
  File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 138 in _clean_done_fetch_futures
  File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 129 in send_fetches
  File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 405 in _message_generator
  File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 560 in __next__
  File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/group.py", line 925 in _message_generator
  File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/group.py", line 964 in __next__
  File "/home/kirill/.PyCharmCE2016.3/config/scratches/scratch_5.py", line 27 in <module>

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

While running

from kafka import KafkaConsumer

for msg in KafkaConsumer(in_topic,
                         bootstrap_servers=[server],
                         auto_offset_reset='earliest',
                         group_id=None,
                         enable_auto_commit=False):
    print(msg)

[dpkp]

You'll need to switch to python-lz4 . This crash is caused by a bug in lz4tools. On Mar 23, 2017 2:10 AM, "Kirill Ignatev" <notifications@github.com> wrote: Python 3.5.1, kafka-python 1.3.3. Problem arises when consuming a lot of messages (more than 15000 in the queue). Can this be caused by #1024 <#1024> ? In kafka-python 1.3.2 there was no such error, but huge memory consumption. Fatal Python error: deallocating None Current thread 0x00007f7411669700 (most recent call first): File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 138 in _clean_done_fetch_futures File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 129 in send_fetches File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 405 in _message_generator File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/fetcher.py", line 560 in __next__ File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/group.py", line 925 in _message_generator File "/home/kirill/f2/lib64/python3.5/site-packages/kafka/consumer/group.py", line 964 in __next__ File "/home/kirill/.PyCharmCE2016.3/config/scratches/scratch_5.py", line 27 in <module> Process finished with exit code 134 (interrupted by signal 6: SIGABRT) While running from kafka import KafkaConsumer for msg in KafkaConsumer(in_topic, bootstrap_servers=[server], auto_offset_reset='earliest', group_id=None, enable_auto_commit=False): print(msg) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1047>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetNV-okyhyAysGdbMX-cV68EShKe0ks5rojcZgaJpZM4MmcUn> .

[kiri11]

Thanks! Updated README #1057