Lazy unpack records in Consumer Fetcher #2555
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(Lots of commits here, planning to split some into separate PRs).
After more closely reviewing fetch_offset code, namely changes from #1724, it looks like there may be edge cases where offset tracking "loses" messages. #2011 highlights this for paused partitions, but I believe the same issue could happen when using the
consumer.poll()
interface and experiencing a rebalance or other event that causes current fetch response data to become "unfetchable". In that case the fetch response will be discarded without returning records, but the subscription position will still be updated to the end of the fetch response! This would cause the discarded records to be "lost" to the consumer without raising an exception or logging any useful message (the prior log message was at level 0, which is lower than debug). I think this issue does not impact the iterator interfacefor msg in consumer:
because we handle lazy offset updates in the iterator.Fix #2011, #2072, #2173, #2310