Implement asynchronous send queue

[dpkp]

One big difference between our networking stack and the one in the java client is that we send requests immediately while the java client puts them on an internal queue (per connection) and tries to send pending requests during calls to poll() . I would like to investigate using the same approach.

[jeffwidman]

Is this for the consumer only?
Does this apply for all API calls?
What are the benefits of switching to this model?

If it's just consumer API calls, is the volume of calls sufficient to justify the increased complexity of doing this asynchronously?

[dpkp]

This would be for both producer and consumer -- likely implemented in KafkaClient and BrokerConnection . The two expected benefits would be (1) better network pipelining, though this probably is more of a benefit for the producer than consumer; (2) allow queuing up sends before connection is complete, which avoids a lot of the NodeNotReady checking / complexity.

[dpkp]

I believe that this is needed to improve client performance, particularly in the consumer. There seem to be several reports of "slow" consumers, and I believe they may be caused by a synchronous send call waiting to acquire the client lock while some other thread is blocked waiting for bytes to read from network I/O in client->poll->select . The select can block for up to the full request_timeout_ms when there is no incoming network traffic, and in cases where there are no in flight requests... there will be no incoming network traffic until the send operation is able to complete. (to fix this we also might check to see if there are any in flight requests and simply skip the select if not... though we would also need to do some other work to prevent high CPU churn during idle periods)