Consumer lag alerts that never fire, rebalances that stall a whole service, offsets that jump backwards after a deploy — most of the consumer-group incidents we see in production trace back to a handful of recurring configuration and design mistakes. Here are the ones worth checking before they page you.
1. max.poll.interval.ms shorter than your worst-case batch
The consumer heartbeats from a background thread, so a busy consumer stays "alive" — but if your poll loop takes longer than max.poll.interval.ms (default 5 minutes) to come back for the next batch, the group coordinator assumes the consumer has failed, kicks it out, and triggers a rebalance. The consumer then finishes its batch, tries to commit, and gets CommitFailedException; the records are reprocessed by whoever took over the partition.
The failure mode is insidious because it only appears under load: a batch that takes 30 seconds on a normal day takes 8 minutes during a backfill, and suddenly the group is in a rebalance loop that never drains the lag causing it. Either bound your per-batch work (max.poll.records down from 500) or raise the interval to cover the genuine worst case — and alert on records-lag-max alongside rebalance rate so you see the loop forming.
2. Auto-commit hiding an at-most-once window
enable.auto.commit=true commits the offsets returned by the previous poll on a timer. If your processing is asynchronous — you hand records to a thread pool and immediately poll again — offsets can be committed for records that have not been processed yet. Crash at the wrong moment and those records are gone from your consumer's point of view.
If you need at-least-once, commit manually after processing completes, and commit specific offsets rather than the blanket "everything from the last poll" when you process out of order. If you keep auto-commit, make sure processing is synchronous within the poll loop.
3. Rebalance storms from eager assignment
The classic eager assignors revoke every partition from every member at the start of a rebalance, so a single pod restart in a 40-instance group stops the world. Two fixes ship with modern clients and most teams still run neither:
partition.assignment.strategy=CooperativeStickyAssignorkeeps unaffected assignments in place and only moves what must move.- Static membership (
group.instance.idset per instance) lets a bounced pod rejoin withinsession.timeout.mswithout triggering a rebalance at all — the right tool for rolling deploys on Kubernetes.
Together they turn a deploy from forty rebalances into roughly zero.
4. More consumers than partitions
A partition is the unit of parallelism: with 12 partitions and 20 consumers, 8 consumers sit idle, and scaling the deployment further changes nothing except rebalance time. We regularly meet teams "scaling out" a lagging consumer group whose partition count quietly capped throughput months ago. Check assignment distribution before adding instances; if the cap is real, the fix is a partition increase (mind key ordering) or faster per-record processing.
5. Lag measured in records, not time
records-lag-max of 100,000 is an emergency on a topic doing 100 records/second and background noise on one doing a million. Alerting on raw record lag produces both false alarms and false calm. Track time lag — how old is the newest unprocessed record — either via broker-side kafka-consumer-groups tooling combined with produce timestamps, or an exporter that reports lag in seconds. Time lag is also what your SLOs are actually about: nobody's contract says "fewer than 50,000 records behind."
6. The poison pill that parks a partition
One malformed record that reliably throws will halt its partition forever while the other partitions sail on — total lag looks modest while one key's data quietly stops. Decide the policy up front: dead-letter the record after N failures, or skip-and-log with an alert. "Retry forever" is a decision too, just usually not one anybody made on purpose.
A quick audit list
- Worst-case poll-loop duration measured, and
max.poll.interval.msabove it - Commit strategy stated in the code: at-least-once with manual commits, or deliberately at-most-once
CooperativeStickyAssignor+ static membership on Kubernetes deployments- Consumer count ≤ partition count, checked per group
- Lag alerting in seconds, per group and per topic
- Dead-letter or skip policy for undeserializable and unprocessable records
None of these require exotic tooling — they are configuration and a little design discipline. They are also the difference between a consumer fleet you deploy at 4 p.m. on a Friday and one you don't.