kafka.errors.NoBrokersAvailable exception when running producer example on Mac

[vkroz]

Running single-node Kafka cluster on localhost on Mac (OS X 10.11.6)
Getting error on attempt to instantiate producer

>>> from kafka import KafkaProducer
>>> producer = KafkaProducer(bootstrap_servers=['localhost:9092'])

getting error

  File "<stdin>", line 1, in <module>
  File "/Users/user1/anaconda/envs/myenv/lib/python2.7/site-packages/kafka/producer/kafka.py", line 347, in __init__
    **self.config)
  File "/Users/user1/anaconda/envs/myenv/lib/python2.7/site-packages/kafka/client_async.py", line 221, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/Users/user1/anaconda/envs/myenv/lib/python2.7/site-packages/kafka/client_async.py", line 826, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Kafka is up, and running locally and producer from confluent-kafka-python works without issues. Any suggestions what to look for?

server.properties:
. . . 
listeners=PLAINTEXT://localhost:9092
. . .

[KIC]

I have the same problem on windows using Kafka 1.0.0

[jeffwidman]

@vkroz what version of kafka-python / kafka brokers?

Can you share the code snippet you used for confluent-kafka-python to connect?

[vkroz]

@jeffwidman It's just 2 lines of code in Python console as shown above.

And apparently it has something to do with setup of specific cluster. When I'm trying to connect to another remote Kafka cluster it works fine.

[jeffwidman]

Are there differences between the server.properties of the two clusters? I'd start by checking the listeners and advertised.listeners configs.

[vkroz]

Here are server properties used to run 3-nodes cluster on localhost -- the one which is python client os failing to connect

https://gist.github.com/vkroz/5ee7854448fdd96ab619a7f7380fa0c2

And these config are fully operations, cluster is accessible by Java producers/consumers and all standard Kafka tools.

[jeffwidman]

These look fine to me, nothing amiss. Can you share the snippet you use to connect using confluent-kafka-python? If you're running from the exact same machine with the same broker address, it's weird to me that the confluent client can connect but kafka-python can't. Are you perhaps running the confluent/java codes from a different container/VM than where you run kafka-python?

[dpkp]

what does this show:

python -c 'import socket; print socket.getaddrinfo("localhost", 9092)'

[jar349]

Version 1.3.5 of this library (which is latest on pypy) only lists certain API versions 0.8.0 to 0.10.1. So unless you explicitly specify api_version to be (0, 10, 1) the client library's attempt to discover the version will cause a NoBrokersAvailable error.

an example from my code:

producer = KafkaProducer(
    bootstrap_servers=URL,
    client_id=CLIENT_ID,
    value_serializer=JsonSerializer.serialize,
    api_version=(0, 10, 1)
)

[dpkp]

Although setting api_version may appear to fix a problem, this is a very wrong assessment:

Version 1.3.5 of this library (which is latest on pypy) only lists certain API versions 0.8.0 to 0.10.1. So unless you explicitly specify api_version to be (0, 10, 1) the client library's attempt to discover the version will cause a NoBrokersAvailable error.

The issue is not the version check, it is the TCP socket connection itself. If kafka-python can connect to a 1.0.0 broker it would still be identified as (0, 10, 1). The only thing you achieve by setting an api_version explicitly is you never actually try to open the TCP socket. But that probably just means that your connection issue will surface later when you try to send or receive messages.

The deeper issue here is a real connection bug that I believe has to do with using all available dns lookup data during bootstrap.

[jar349]

When I explicitly set the api_version, I am able to produce and consume events.

[dpkp]

I believe the underlying issue is related to handling multiple network interfaces. My mac, for example, shows three addrinfos for localhost:

('::1', 9092, 0, 0)
('fe80::1%lo0', 9092, 0, 1)
('127.0.0.1', 9092)

The bootstrap and version check code currently only attempt connection on the first addrinfo returned, so if the broker is bound to a later addrinfo then the connection will fail. I fixed this in #1411 .

[jar349]

May I please ask why explicitly setting the api_version always works for me but removing it always fails? It doesn't seem to me that the value of api_version would in any way effect which network interface is bound?

[dpkp]

When you set api_version the client will not attempt to probe brokers for version information. So it is the probe operation that is failing. One large difference between the version probe connections and the general connections is that the former only attempts to connect on a single interface per connection (per broker), where as the latter -- general operation -- will cycle through all interfaces continually until a connection succeeds. #1411 fixes this by switching the version probe logic to attempt a connection on all found interfaces.

[jar349]

That makes perfect sense, thank you! 👍