KeyError Creating SimpleProducer

[herriojr]

This is an issue I've been experiencing running Master branch. Basically, the first time I access an uncreated topic, I get a KeyError, whereas the second time, I can access it just fine.

import kafka
client = kafka.client.KafkaClient(address, port)
producer = kafka.producer.SimpleProducer(client, "test") # Fails first time called, succeeds thereafter

Traceback (most recent call last):
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1532, in
debugger.run(setup['file'], None, None)
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1143, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "/projects/scratch/init.py", line 27, in
main()
File "/home/vagrant/.virtualenvs/app/local/lib/python2.7/site-packages/kafka/producer.py", line 191, in init
self.next_partition = cycle(client.topic_partitions[topic])
KeyError: 'test'

[rdiomar]

Not sure why, but when it's a new topic kafka doesn't return the partition info to the first metadata request. As a workaround, you can do what ensure_topic_creation() in test_integration.py does before creating your producer. We should fix this though.

[dpkp]

Latest master (after merging #111 pull request) changes behavior so the Producer init no longer takes a topic. But this error could still show up on first message send to a new topic. Can you test?

[dpkp]

My testing suggests that when topic auto-creation is enabled on the kafka server (auto.create.topics.enable=True), it will create the topic but return a LeaderNotAvailable error while it does the initial setup asynchronously. this means that the message produce has to wait until the topic is fully setup w/ a broker leader. kafka-python should probably check for this error and optionally retry (w/ resyncing leader metadata?)

[hmahmood]

Any updates on this?

[frgtn]

As of 8811298 this still breaks with KeyError:
Traceback (most recent call last):

  File "test_113.py", line 21, in <module>
    'The message')
  File "<some_prefix>/lib/python2.7/site-packages/kafka/producer.py", line 204, in send_messages
    partition = self._next_partition(topic)
  File "<_some_prefix>/lib/python2.7/site-packages/kafka/producer.py", line 200, in _next_partition
    self.partition_cycles[topic] = cycle(self.client.topic_partitions[topic])
KeyError: 'test_2'

[dpkp]

merged #109 -- please retry w/ master trunk

[imduffy15]

Still seeing this issue on master.

Any updates?

[wizzat]

I can confirm the issue happening, and I've looked into it enough to know why. Right now I am working on beefing up the test suite.

On a scale of 1-10, how much of a blocker is this for you?

-Mark

On Apr 9, 2014, at 3:23, Ian Duffy notifications@github.com wrote:

Still seeing this issue on master.

Any updates?

—
Reply to this email directly or view it on GitHub.

[imduffy15]

I'm working around it for the moment by creating the topics manually.

Lets go with 4 - Annoying.
On Apr 9, 2014 5:53 PM, "Mark Roberts" notifications@github.com wrote:

I can confirm the issue happening, and I've looked into it enough to know
why. Right now I am working on beefing up the test suite.

On a scale of 1-10, how much of a blocker is this for you?

-Mark

On Apr 9, 2014, at 3:23, Ian Duffy notifications@github.com wrote:

Still seeing this issue on master.

Any updates?

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/113#issuecomment-39988130
.

[Korijn]

I'm getting exactly this error constantly, not just on the first try. Not sure why yet.

Traceback (most recent call last):
File "C:.........\logging\handlers\kafkahandler.py", line 42, in emit
self.producer.send_messages(self.topic, msg)
File "C:.........\VirtualEnv\lib\site-packages\kafka\producer.py", line 204, in send_messages
partition = self._next_partition(topic)
File "C:.........\VirtualEnv\lib\site-packages\kafka\producer.py", line 200, in _next_partition
self.partition_cycles[topic] = cycle(self.client.topic_partitions[topic])
KeyError: '.........'

[wizzat]

Do you have a topic without ISR or without a leader? Can you print
kafka-list-topics output?

-Mark

On Tue, May 20, 2014 at 4:22 AM, Korijn van Golen
notifications@github.comwrote:

I'm getting exactly this error constantly, not just on the first try. Not
sure why yet.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/113#issuecomment-43613360
.

[Korijn]

I feel silly. It was a configuration issue. The kafka-server-start.sh and zookeeper-server-start.sh scripts that come with the packaged install from the ubuntu repo's were not exactly suitable for use with supervisord. After correcting setup all problems were solved. I can post the scripts if you're interested.

[dpkp]

Nonetheless, there's an assumption in the code that we'll always get topic metadata back from the server. We need to handle this more gracefully.

https://github.com/mumrah/kafka-python/blob/master/kafka/producer.py#L219