consumer deadlock happen after log “Heartbeat: local member_id was not recognized; this consumer needs to re-join”

[zhgjun]

I use kafka-python 1.4.4, enable_auto_commit=False. One day I found my consumer hang and cannot receive msg.
the log is:

2018-10-29T16:12:44.361+08:00 localhost WARNING [pid:5273] [MainThread][tid:65330864] [base.py:822 _handle_heartbeat_response] Heartbeat: local member_id was not recognized; this consumer needs to re-join
2018-10-29T16:12:44.371+08:00 localhost  INFO [pid:5273] [MainThread][tid:65330864] [consumer.py:341 _on_join_prepare] Revoking previously assigned partitions set([TopicPartition(topic=u'agent', partition=2), TopicPartition(topic=u'agent', partition=3), TopicPartition(topic=u'agent', partition=4), TopicPartition(topic=u'agent', partition=5), TopicPartition(topic=u'agent', partition=6), TopicPartition(topic=u'agent', partition=7), TopicPartition(topic=u'agent', partition=0), TopicPartition(topic=u'agent', partition=1)]) for group agent

and I got the stack traces:

stack 1: the heartbeat thread

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/coordinator/base.py:914 in run
    'self._run_once()'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/coordinator/base.py:982 in _run_once
    'with self.coordinator._client._lock, self.coordinator._lock:'

/usr/lib64/python2.7/threading.py:285 in __enter__
    'return self.__lock.__enter__()'

/usr/lib64/python2.7/threading.py:173 in acquire
    'rc = self.__block.acquire(blocking)'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/eventlet/semaphore.py:113 in acquire
    'hubs.get_hub().switch()'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/eventlet/hubs/hub.py:294 in switch
    'return self.greenlet.switch()'

stack 2: poll msg thread

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/consumer/group.py:609 in poll
    'records = self._poll_once(remaining, max_records)'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/consumer/group.py:629 in _poll_once
    'self._coordinator.poll()'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/coordinator/consumer.py:281 in poll
    'self.ensure_active_group()'  **(ps: it will get the self.coordinator._lock )**

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/coordinator/base.py:373 in ensure_active_group
    'while not self.coordinator_unknown():'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/coordinator/base.py:227 in coordinator_unknown
    'return self.coordinator() is None'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/coordinator/base.py:236 in coordinator
    'elif self._client.is_disconnected(self.coordinator_id):'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/client_async.py:433 in is_disconnected
    'with self._lock:' **(ps: self._lock is self.coordinator._client._lock too)**

/usr/lib64/python2.7/threading.py:173 in acquire
    'rc = self.__block.acquire(blocking)'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/eventlet/semaphore.py:113 in acquire
    'hubs.get_hub().switch()'

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/eventlet/hubs/hub.py:294 in switch
    'return self.greenlet.switch()'

I think the reason maybe :

1.when stack 2 get the self.coordinator._lock first

def ensure_active_group(self):
        """Ensure that the group is active (i.e. joined and synced)"""
        with self._lock:
            if self._heartbeat_thread is None:
                self._start_heartbeat_thread()

2.and then it will acquire the self.coordinator._client._lock

/opt/cloud/services/network-agent/venv/lib/python2.7/site-packages/kafka/client_async.py:433 in is_disconnected
    'with self._lock:'

3.but at this time stack 1 got self.coordinator._client._lock and acquire self.coordinator._lock

with self.coordinator._client._lock, self.coordinator._lock:

4.and finally dead lock happen.

[zhgjun]

I also think it maybe same as this issue #1544

[zhgjun]

I find the issue #1464 (which fixed by @dpkp )related to this problem, and I think we can fix it by (add acquire self._client._lock)

def ensure_active_group(self):
        """Ensure that the group is active (i.e. joined and synced)"""
        with self._client._lock, self._lock:
            if self._heartbeat_thread is None:
                self._start_heartbeat_thread()

[licy121]

I also found this deadlock problem.
help? / @dpkp / @tvoinarovskyi / @jeffwidman

[vimal3271]

@zhgjun did you see this issue in 1.4.3 or 1.4.4 ? Because the fix went in 1.4.4. We are seeing some consumer hangs in 1.4.3 and planning to upgrade to 1.4.4. It must be a typo since the release of 1.4.4 is only on nov 23 , 2018.

[zhgjun]

@zhgjun did you see this issue in 1.4.3 or 1.4.4 ? Because the fix went in 1.4.4. We are seeing some consumer hangs in 1.4.3 and planning to upgrade to 1.4.4. It must be a typo since the release of 1.4.4 is only on nov 23 , 2018.

I think I have check the diff between 1.4.2 and 1.4.3. I believe that this problem also exists. so I recommend upgrade to 1.4.4 and synchronize the bug fixes on the master branch. such like the bug
#1695

[dnascimb]

Still seeing similar behavior to this in version 1.4.6.

{"message": {"short_msg": "Updated partition assignment: [TopicPartition(topic='alldata_address', partition=0), TopicPartition(topic='alldata_address', partition=1), TopicPartition(topic='alldata_address', partition=2), TopicPartition(topic='alldata_address', partition=3), TopicPartition(topic='alldata_address', partition=4), TopicPartition(topic='alldata_address', partition=5), TopicPartition(topic='alldata_address', partition=6), TopicPartition(topic='alldata_address', partition=7), TopicPartition(topic='alldata_address', partition=8), TopicPartition(topic='alldata_address', partition=9), TopicPartition(topic='alldata_address', partition=10), TopicPartition(topic='alldata_address', partition=11), TopicPartition(topic='alldata_auction', partition=0), TopicPartition(topic='alldata_auction', partition=1), TopicPartition(topic='alldata_auction', partition=2), TopicPartition(topic='alldata_auction', partition=3), TopicPartition(topic='alldata_auction', partition=4), TopicPartition(topic='alldata_auction', partition=5), TopicPartition(topic='alldata_auction', partition=6), TopicPartition(topic='alldata_auction', partition=7), TopicPartition(topic='alldata_auction', partition=8), TopicPartition(topic='alldata_auction', partition=9), TopicPartition(topic='alldata_auction', partition=10), TopicPartition(topic='alldata_auction', partition=11), TopicPartition(topic='alldata_bid', partition=0), TopicPartition(topic='alldata_bid', partition=1), TopicPartition(topic='alldata_bid', partition=2), TopicPartition(topic='alldata_bid', partition=3), TopicPartition(topic='alldata_bid', partition=4), TopicPartition(topic='alldata_bid', partition=5), TopicPartition(topic='alldata_bid', partition=6), TopicPartition(topic='alldata_bid', partition=7), TopicPartition(topic='alldata_bid', partition=8), TopicPartition(topic='alldata_bid', partition=9), TopicPartition(topic='alldata_bid', partition=10), TopicPartition(topic='alldata_bid', partition=11), TopicPartition(topic='alldata_category', partition=0), TopicPartition(topic='alldata_category', partition=1), TopicPartition(topic='alldata_category', partition=2), TopicPartition(topic='alldata_category', partition=3), TopicPartition(topic='alldata_category', partition=4), TopicPartition(topic='alldata_category', partition=5), TopicPartition(topic='alldata_category', partition=6), TopicPartition(topic='alldata_category', partition=7), TopicPartition(topic='alldata_category', partition=8), TopicPartition(topic='alldata_category', partition=9), TopicPartition(topic='alldata_category', partition=10), TopicPartition(topic='alldata_category', partition=11), TopicPartition(topic='alldata_customer', partition=0), TopicPartition(topic='alldata_customer', partition=1), TopicPartition(topic='alldata_customer', partition=2), TopicPartition(topic='alldata_customer', partition=3), TopicPartition(topic='alldata_customer', partition=4), TopicPartition(topic='alldata_customer', partition=5), TopicPartition(topic='alldata_customer', partition=6), TopicPartition(topic='alldata_customer', partition=7), TopicPartition(topic='alldata_customer', partition=8), TopicPartition(topic='alldata_customer', partition=9), TopicPartition(topic='alldata_customer', partition=10), TopicPartition(topic='alldata_customer', partition=11), TopicPartition(topic='alldata_order', partition=0), TopicPartition(topic='alldata_order', partition=1), TopicPartition(topic='alldata_order', partition=2), TopicPartition(topic='alldata_order', partition=3), TopicPartition(topic='alldata_order', partition=4), TopicPartition(topic='alldata_order', partition=5), TopicPartition(topic='alldata_order', partition=6), TopicPartition(topic='alldata_order', partition=7), TopicPartition(topic='alldata_order', partition=8), TopicPartition(topic='alldata_order', partition=9), TopicPartition(topic='alldata_order', partition=10), TopicPartition(topic='alldata_order', partition=11), TopicPartition(topic='alldata_product', partition=0), TopicPartition(topic='alldata_product', partition=1), TopicPartition(topic='alldata_product', partition=2), TopicPartition(topic='alldata_product', partition=3), TopicPartition(topic='alldata_product', partition=4), TopicPartition(topic='alldata_product', partition=5), TopicPartition(topic='alldata_product', partition=6), TopicPartition(topic='alldata_product', partition=7), TopicPartition(topic='alldata_product', partition=8), TopicPartition(topic='alldata_product', partition=9), TopicPartition(topic='alldata_product', partition=10), TopicPartition(topic='alldata_product', partition=11), TopicPartition(topic='alldata_recent_auction', partition=0), TopicPartition(topic='alldata_recent_auction', partition=1), TopicPartition(topic='alldata_recent_auction', partition=2), TopicPartition(topic='alldata_recent_auction', partition=3), TopicPartition(topic='alldata_recent_auction', partition=4), TopicPartition(topic='alldata_recent_auction', partition=5), TopicPartition(topic='alldata_recent_auction', partition=6), TopicPartition(topic='alldata_recent_auction', partition=7), TopicPartition(topic='alldata_recent_auction', partition=8), TopicPartition(topic='alldata_recent_auction', partition=9), TopicPartition(topic='alldata_recent_auction', partition=10), TopicPartition(topic='alldata_recent_auction', partition=11), TopicPartition(topic='alldata_seller', partition=0), TopicPartition(topic='alldata_seller', partition=1), TopicPartition(topic='alldata_seller', partition=2), TopicPartition(topic='alldata_seller', partition=3), TopicPartition(topic='alldata_seller', partition=4), TopicPartition(topic='alldata_seller', partition=5), TopicPartition(topic='alldata_seller', partition=6), TopicPartition(topic='alldata_seller', partition=7), TopicPartition(topic='alldata_seller', partition=8), TopicPartition(topic='alldata_seller', partition=9), TopicPartition(topic='alldata_seller', partition=10), TopicPartition(topic='alldata_seller', partition=11), TopicPartition(topic='alldata_watchlist', partition=0), TopicPartition(topic='alldata_watchlist', partition=1), TopicPartition(topic='alldata_watchlist', partition=2), TopicPartition(topic='alldata_watchlist', partition=3), TopicPartition(topic='alldata_watchlist', partition=4), TopicPartition(topic='alldata_watchlist', partition=5), TopicPartition(topic='alldata_watchlist', partition=6), TopicPartition(topic='alldata_watchlist', partition=7), TopicPartition(topic='alldata_watchlist', partition=8), TopicPartition(topic='alldata_watchlist', partition=9), TopicPartition(topic='alldata_watchlist', partition=10), TopicPartition(topic='alldata_watchlist', partition=11)]"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "INFO", "level": 6, "log_name": "kafka.consumer.subscription_state", "timestamp": "2019-04-19T01:12:16.674164Z"} {"message": {"short_msg": "Setting newly assigned partitions {TopicPartition(topic='alldata_watchlist', partition=4), TopicPartition(topic='alldata_order', partition=1), TopicPartition(topic='alldata_category', partition=3), TopicPartition(topic='alldata_seller', partition=0), TopicPartition(topic='alldata_watchlist', partition=9), TopicPartition(topic='alldata_bid', partition=2), TopicPartition(topic='alldata_recent_auction', partition=10), TopicPartition(topic='alldata_address', partition=5), TopicPartition(topic='alldata_customer', partition=2), TopicPartition(topic='alldata_auction', partition=4), TopicPartition(topic='alldata_category', partition=4), TopicPartition(topic='alldata_product', partition=2), TopicPartition(topic='alldata_recent_auction', partition=5), TopicPartition(topic='alldata_order', partition=9), TopicPartition(topic='alldata_seller', partition=8), TopicPartition(topic='alldata_address', partition=8), TopicPartition(topic='alldata_customer', partition=7), TopicPartition(topic='alldata_category', partition=9), TopicPartition(topic='alldata_watchlist', partition=3), TopicPartition(topic='alldata_bid', partition=8), TopicPartition(topic='alldata_product', partition=7), TopicPartition(topic='alldata_recent_auction', partition=0), TopicPartition(topic='alldata_order', partition=4), TopicPartition(topic='alldata_seller', partition=5), TopicPartition(topic='alldata_bid', partition=5), TopicPartition(topic='alldata_product', partition=10), TopicPartition(topic='alldata_address', partition=0), TopicPartition(topic='alldata_watchlist', partition=6), TopicPartition(topic='alldata_auction', partition=1), TopicPartition(topic='alldata_order', partition=3), TopicPartition(topic='alldata_seller', partition=2), TopicPartition(topic='alldata_category', partition=1), TopicPartition(topic='alldata_watchlist', partition=11), TopicPartition(topic='alldata_bid', partition=0), TopicPartition(topic='alldata_recent_auction', partition=8), TopicPartition(topic='alldata_address', partition=7), TopicPartition(topic='alldata_customer', partition=4), TopicPartition(topic='alldata_auction', partition=6), TopicPartition(topic='alldata_category', partition=10), TopicPartition(topic='alldata_bid', partition=11), TopicPartition(topic='alldata_product', partition=4), TopicPartition(topic='alldata_recent_auction', partition=3), TopicPartition(topic='alldata_address', partition=10), TopicPartition(topic='alldata_auction', partition=9), TopicPartition(topic='alldata_order', partition=11), TopicPartition(topic='alldata_seller', partition=10), TopicPartition(topic='alldata_customer', partition=9), TopicPartition(topic='alldata_watchlist', partition=5), TopicPartition(topic='alldata_order', partition=6), TopicPartition(topic='alldata_seller', partition=7), TopicPartition(topic='alldata_category', partition=2), TopicPartition(topic='alldata_bid', partition=3), TopicPartition(topic='alldata_recent_auction', partition=11), TopicPartition(topic='alldata_address', partition=2), TopicPartition(topic='alldata_customer', partition=1), TopicPartition(topic='alldata_auction', partition=3), TopicPartition(topic='alldata_category', partition=7), TopicPartition(topic='alldata_bid', partition=7), TopicPartition(topic='alldata_product', partition=1), TopicPartition(topic='alldata_recent_auction', partition=6), TopicPartition(topic='alldata_address', partition=9), TopicPartition(topic='alldata_customer', partition=6), TopicPartition(topic='alldata_category', partition=8), TopicPartition(topic='alldata_watchlist', partition=0), TopicPartition(topic='alldata_bid', partition=9), TopicPartition(topic='alldata_auction', partition=11), TopicPartition(topic='alldata_order', partition=5), TopicPartition(topic='alldata_seller', partition=4), TopicPartition(topic='alldata_product', partition=6), TopicPartition(topic='alldata_recent_auction', partition=1), TopicPartition(topic='alldata_bid', partition=6), TopicPartition(topic='alldata_product', partition=9), TopicPartition(topic='alldata_customer', partition=11), TopicPartition(topic='alldata_address', partition=1), TopicPartition(topic='alldata_watchlist', partition=7), TopicPartition(topic='alldata_auction', partition=0), TopicPartition(topic='alldata_order', partition=0), TopicPartition(topic='alldata_category', partition=0), TopicPartition(topic='alldata_seller', partition=1), TopicPartition(topic='alldata_watchlist', partition=8), TopicPartition(topic='alldata_bid', partition=1), TopicPartition(topic='alldata_recent_auction', partition=9), TopicPartition(topic='alldata_address', partition=4), TopicPartition(topic='alldata_customer', partition=3), TopicPartition(topic='alldata_auction', partition=5), TopicPartition(topic='alldata_category', partition=5), TopicPartition(topic='alldata_product', partition=3), TopicPartition(topic='alldata_auction', partition=8), TopicPartition(topic='alldata_order', partition=8), TopicPartition(topic='alldata_seller', partition=9), TopicPartition(topic='alldata_recent_auction', partition=4), TopicPartition(topic='alldata_address', partition=11), TopicPartition(topic='alldata_customer', partition=8), TopicPartition(topic='alldata_watchlist', partition=2), TopicPartition(topic='alldata_order', partition=7), TopicPartition(topic='alldata_seller', partition=6), TopicPartition(topic='alldata_bid', partition=4), TopicPartition(topic='alldata_product', partition=11), TopicPartition(topic='alldata_address', partition=3), TopicPartition(topic='alldata_customer', partition=0), TopicPartition(topic='alldata_auction', partition=2), TopicPartition(topic='alldata_order', partition=2), TopicPartition(topic='alldata_seller', partition=3), TopicPartition(topic='alldata_category', partition=6), TopicPartition(topic='alldata_watchlist', partition=10), TopicPartition(topic='alldata_product', partition=0), TopicPartition(topic='alldata_recent_auction', partition=7), TopicPartition(topic='alldata_address', partition=6), TopicPartition(topic='alldata_customer', partition=5), TopicPartition(topic='alldata_auction', partition=7), TopicPartition(topic='alldata_category', partition=11), TopicPartition(topic='alldata_watchlist', partition=1), TopicPartition(topic='alldata_bid', partition=10), TopicPartition(topic='alldata_product', partition=5), TopicPartition(topic='alldata_recent_auction', partition=2), TopicPartition(topic='alldata_auction', partition=10), TopicPartition(topic='alldata_order', partition=10), TopicPartition(topic='alldata_seller', partition=11), TopicPartition(topic='alldata_product', partition=8), TopicPartition(topic='alldata_customer', partition=10)} for group pygs-consumer"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "INFO", "level": 6, "log_name": "kafka.coordinator.consumer", "timestamp": "2019-04-19T01:12:16.674926Z"} {"message": {"short_msg": "flushing due to time since last flush 10.847s > max_flush_time 10.847s"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "elasticapm.transport", "timestamp": "2019-04-19T01:12:20.296575Z"} {"message": {"short_msg": "Heartbeat session expired, marking coordinator dead"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "WARNING", "level": 4, "log_name": "kafka.coordinator", "timestamp": "2019-04-19T01:12:26.725351Z"} {"message": {"short_msg": "Marking the coordinator dead (node 7) for group pygs-consumer: Heartbeat session expired."}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "WARNING", "level": 4, "log_name": "kafka.coordinator", "timestamp": "2019-04-19T01:12:26.726657Z"} {"message": {"short_msg": "flushing due to time since last flush 10.119s > max_flush_time 10.118s"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "elasticapm.transport", "timestamp": "2019-04-19T01:12:30.415634Z"} {"message": {"short_msg": "Starting metrics collect timer"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "elasticapm.metrics", "timestamp": "2019-04-19T01:12:31.459706Z"} {"message": {"short_msg": "Collecting metrics"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "elasticapm.metrics", "timestamp": "2019-04-19T01:12:31.460479Z"} {"message": {"short_msg": "flushing due to time since last flush 9.227s > max_flush_time 9.227s"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "elasticapm.transport", "timestamp": "2019-04-19T01:12:39.643333Z"} {"message": {"short_msg": "Resetting dropped connection: logs-apm-dev.bstock.io"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "urllib3.connectionpool", "timestamp": "2019-04-19T01:12:39.644201Z"} {"message": {"short_msg": "https://logs-apm-dev.bstock.io:5222 \"POST /intake/v2/events HTTP/1.1\" 202 0"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "DEBUG", "level": 7, "log_name": "urllib3.connectionpool", "timestamp": "2019-04-19T01:12:39.672407Z"} {"message": {"short_msg": "Sent request, url=https://logs-apm-dev.bstock.io:5222/intake/v2/events size=0.44kb status=202"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "INFO", "level": 6, "log_name": "elasticapm.transport.http", "timestamp": "2019-04-19T01:12:39.672901Z"} {"message": {"short_msg": "Group coordinator for pygs-consumer is BrokerMetadata(nodeId=7, host='b7-pkc-4v78z.us-west1.gcp.confluent.cloud', port=9092, rack=None)"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "INFO", "level": 6, "log_name": "kafka.cluster", "timestamp": "2019-04-19T01:12:44.064389Z"} {"message": {"short_msg": "Discovered coordinator 7 for group pygs-consumer"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "INFO", "level": 6, "log_name": "kafka.coordinator", "timestamp": "2019-04-19T01:12:44.064812Z"} {"message": {"short_msg": "Heartbeat: local member_id was not recognized; this consumer needs to re-join"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "WARNING", "level": 4, "log_name": "kafka.coordinator", "timestamp": "2019-04-19T01:12:44.859616Z"} {"message": {"short_msg": "Heartbeat: local member_id was not recognized; this consumer needs to re-join"}, "context": {"app_name": "alldata-consumer", "env": "TEST"}, "level_name": "WARNING", "level": 4, "log_name": "kafka.coordinator", "timestamp": "2019-04-19T01:12:44.860203Z"}

The consumer just keeps joining the consumer group, getting topic/partition assignments, and encountering Heartbeat session expired, marking coordinator dead, Marking the coordinator dead (node 7) for group pygs-consumer: Heartbeat session expired., and then Heartbeat: local member_id was not recognized; this consumer needs to re-join.

Here is the consumer connection code:

            self.logger.info('Connecting to kafka',
                             host=self.kafka_bootstrap_host)
            kafka_consumer = KafkaConsumer(
                bootstrap_servers=self.kafka_bootstrap_host,
                group_id=self.GROUP_ID,
                key_deserializer=bytes.decode,
                value_deserializer=json.loads,
                enable_auto_commit=False,
                max_poll_records=10,
                consumer_timeout_ms=500,
                metadata_max_age_ms=3000
            )
            kafka_consumer.subscribe(self.TOPIC_LIST)
            self.logger.info("Connected to kafka",
                             host=self.kafka_bootstrap_host)
            return kafka_consumer```

[jeffwidman]

@dnascimb please open a new ticket, as the underlying root cause of this one appears to be fixed.

[tercel]

2020-06-23 11:12:42,644 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:982 Heartbeat session expired, marking coordinator dead
2020-06-23 11:12:42,644 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:717 Marking the coordinator dead (node coordinator-100) for group risk-apilog: Heartbeat session expired.
2020-06-23 11:12:44,307 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:840 Heartbeat: local member_id was not recognized; this consumer needs to re-join
2020-06-23 11:12:44,384 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:840 Heartbeat: local member_id was not recognized; this consumer needs to re-join

[tercel]

Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import kafka
kafka.version
'2.0.1'

2020-06-23 11:12:42,644 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:982 Heartbeat session expired, marking coordinator dead
2020-06-23 11:12:42,644 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:717 Marking the coordinator dead (node coordinator-100) for group risk-apilog: Heartbeat session expired.
2020-06-23 11:12:44,307 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:840 Heartbeat: local member_id was not recognized; this consumer needs to re-join
2020-06-23 11:12:44,384 WARNING /usr/local/lib/python3.7/site-packages/kafka/coordinator/base.py:840 Heartbeat: local member_id was not recognized; this consumer needs to re-join