Closed
Description
KafkaConsumer.poll()
is effectively busy waiting, as KafkaConsumer._poll_once()
doesn't respect timeout_ms
.
_poll_once()
sets timeout_ms
of KafkaClient.poll()
to:
min(timeout_ms, self._coordinator.time_to_next_poll())
However, ConsumerCoordinator.time_to_next_poll()
returns the time in seconds, not milliseconds. The effective timeout is always much lower than timeout_ms
, causing KafkaConsumer.poll()
to spin, consuming a lot of CPU.
Metadata
Metadata
Assignees
Labels
No labels