Most Kafka teams have a rebalance story: a rolling deploy that stalled a consumer group for ninety seconds, a group that spent an afternoon bouncing because one instance kept timing out. The classic rebalance protocol earned those stories honestly — it is a group-wide, stop-the-world agreement with a single point of coordination, the group leader, sitting inside one of your consumer processes.
KIP-848, generally available for consumers in Kafka 4.0, replaces that design. It is the largest change to consumer-group mechanics since the protocol was written, and it arrives through a config flag rather than a rewrite. The migration is genuinely incremental, but it is not free of edges. Here is what changes and the order we roll it out.
What the new protocol actually changes
Assignment moves to the broker. Under the classic protocol, the group coordinator picks a leader from among the members, ships it the membership list, and waits for that leader to compute and return an assignment through a SyncGroup round trip. Under the new protocol the group coordinator computes the assignment itself. There is no group leader, which removes an entire failure class: a slow or wedged leader can no longer hold up the group.
The global barrier is gone. Classic rebalances are an all-members-stop, all-members-rejoin event. KIP-848 reconciles incrementally over heartbeats: the coordinator tells each member its target assignment, each member revokes what it must and acknowledges, and the coordinator hands out the freed partitions as they become available. Members that keep their partitions never stop consuming. In practice, a rolling restart stops looking like a series of cliff edges in your lag graph and starts looking like a ripple.
Heartbeats do the whole job. ConsumerGroupHeartbeat carries membership, subscription, and assignment state in one RPC. JoinGroup and SyncGroup are not part of the consumer path anymore. Reconciliation is driven by heartbeat cadence, so a member that is late to acknowledge delays only the partitions it holds.
Timeouts become group configuration. This is the change that surprises people. Under group.protocol=consumer, session.timeout.ms, heartbeat.interval.ms, and partition.assignment.strategy are no longer client-side knobs — the broker owns them as group.consumer.session.timeout.ms, group.consumer.heartbeat.interval.ms, and the server-side assignor list. Clients that set the old properties will see them rejected or ignored depending on version. Assignor choice is expressed per-client with group.remote.assignor (the uniform and range assignors ship in the box), and the operator bounds what is allowed.
One thing does not move: max.poll.interval.ms is still client-side, still the limit on how long your poll loop may take, and still the cause of more evictions than any other setting. The new protocol does not rescue a slow processing loop.
What stays the same
Offsets, __consumer_offsets, auto.offset.reset, commit semantics, and the consumer API are unchanged. Static membership still works — group.instance.id still lets a bounced pod rejoin without moving partitions, and it is still worth setting on Kubernetes. kafka-consumer-groups.sh still describes groups, though its output gains a protocol column. Kafka Streams is a separate story: it keeps its own assignment logic and gets its own protocol under KIP-1071, so a Streams application is not covered by this migration.
Mixed-mode operation, and its one real constraint
Brokers can run both protocols at once, and a single group can contain classic and new-protocol members while you migrate. The coordinator converts between the two views, so a group half-upgraded keeps working. That is what makes a rolling migration possible.
The constraint worth writing on the change ticket: a mixed group falls back to classic behaviour for the members that need it, so you do not get the incremental-reconciliation benefit until the last classic member leaves. Measure the improvement after the group is fully converted, not halfway through, or you will conclude the upgrade did nothing.
A rollout order
- Get the brokers to 4.x and the metadata version up. The new protocol needs the group coordinator that ships with 4.0 and an appropriate
metadata.version. If you are still finalizing a KRaft migration, finish that first; do not run two protocol migrations in the same window. - Confirm the server side is enabled. Check
group.coordinator.rebalance.protocolsincludesconsumer, and set the group-level defaults deliberately —group.consumer.session.timeout.msandgroup.consumer.heartbeat.interval.msnow apply to every group that opts in, so pick values that match your slowest pod startup, not the defaults you inherited. - Upgrade client libraries first, protocol second. Move consumers to a 4.x client while leaving
group.protocolatclassic. This separates "new jar" incidents from "new protocol" incidents, and the client upgrade is the riskier half for most codebases. - Delete the configs that no longer apply. Grep your consumer config for
session.timeout.ms,heartbeat.interval.ms, andpartition.assignment.strategy. Decide the equivalent group-level value for each and record it. Teams that skip this step discover at rollout that a carefully tuned 45-second session timeout silently became the broker default. - Convert one non-critical group. Set
group.protocol=consumeron a single group with real but tolerant traffic. Watch rebalance rate, time lag, and eviction counts across a full deploy cycle before touching anything customer-facing. - Convert the rest, group by group, finishing each one. Leave no group permanently mixed. A half-converted group is a migration state, not an operating state.
- Have a rollback that is one config away. Flipping
group.protocolback toclassicand restarting returns the group to the old behaviour. Keep that reversibility until the group has survived a peak day and a deploy.
What to watch afterwards
Rebalance rate per group should drop, and the duration of consumption gaps during deploys should drop further — that gap is the number your users feel. Watch group coordinator CPU on the brokers: assignment computation moved onto them, and while it is modest per group, an estate with hundreds of large groups shifts real work broker-side. Watch eviction and fence counts too; an max.poll.interval.ms problem that used to hide inside general rebalance noise becomes much more visible once the noise is gone, which is usually a diagnosis rather than a regression.
Is it worth doing now?
If your consumer groups are small, stable, and deployed rarely, the classic protocol will keep working and there is no urgency — though the direction of travel is clear enough that we would not build new tooling around classic-only assumptions. If you run large groups on Kubernetes, deploy several times a day, or have ever had a rebalance storm in an incident review, the payoff is concrete: deploys stop showing up in your lag graph.
The honest framing is that this is an infrastructure change with an operational benefit and no application-code benefit. It should be scheduled like a broker upgrade — inventoried, rehearsed on a staging cluster, rolled out group by group with a rollback in hand — and not like a config tweak someone lands on a Friday. If you want a second pair of eyes on the inventory or the group-level timeout values before you start, that conversation is what we do.