Skip to content

Lazy unpack records in Consumer Fetcher #2555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Mar 18, 2025
Merged

Conversation

dpkp
Copy link
Owner

@dpkp dpkp commented Mar 18, 2025

(Lots of commits here, planning to split some into separate PRs).

After more closely reviewing fetch_offset code, namely changes from #1724, it looks like there may be edge cases where offset tracking "loses" messages. #2011 highlights this for paused partitions, but I believe the same issue could happen when using the consumer.poll() interface and experiencing a rebalance or other event that causes current fetch response data to become "unfetchable". In that case the fetch response will be discarded without returning records, but the subscription position will still be updated to the end of the fetch response! This would cause the discarded records to be "lost" to the consumer without raising an exception or logging any useful message (the prior log message was at level 0, which is lower than debug). I think this issue does not impact the iterator interface for msg in consumer: because we handle lazy offset updates in the iterator.

Fix #2011, #2072, #2173, #2310

@dpkp dpkp merged commit 14643ca into master Mar 18, 2025
17 checks passed
@dpkp dpkp deleted the dpkp/fetcher-next-fetch-offset branch March 18, 2025 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant