Kafka has always been a log, not a queue. That distinction is the reason it scales, and it is also the reason teams keep bolting a queue onto the side of it — because a fair amount of real work is not a stream of ordered events per key. It is a pile of independent jobs: send this email, render this thumbnail, re-score this account. For that shape of work, the consumer group model fights you. One consumer per partition caps parallelism, a single slow record parks a partition, and offsets advance in one direction whether or not each item succeeded.
KIP-932 — share groups, sometimes described as "queues for Kafka" — is Kafka's answer to that. It went generally available in Kafka 4.1 after shipping as an early-access preview in 4.0. It is new broker mechanics rather than a client-side trick, and it is worth understanding before someone reads a headline and proposes replacing every consumer group with one.
What a share group actually is
A share group is a second kind of group that reads the same topics consumer groups do. The difference is in what the broker tracks.
Many consumers can read one partition. A share group does not assign partitions exclusively. Several members can fetch from the same partition at the same time, and the broker hands each of them a different set of records. Your consumer count is no longer bounded by your partition count — the hardest constraint in the consumer-group model simply does not apply.
Records are acknowledged individually. Instead of one committed offset per partition, the broker maintains per-record delivery state: available, acquired, acknowledged, or archived. When a consumer fetches a record it is acquired for that consumer for the duration of an acquisition lock. The consumer then acknowledges with one of three dispositions — ACCEPT, RELEASE (make it available again immediately), or REJECT (archive it, do not redeliver).
Unacknowledged records come back. If the lock expires before acknowledgement, the record returns to available and goes to someone else. A per-record delivery counter tracks attempts, and once it exceeds the configured limit the record is archived. That is a redelivery-and-give-up policy the broker enforces, rather than a dead-letter pattern each team reimplements.
The client API is KafkaShareConsumer. It looks familiar but is not a drop-in KafkaConsumer: no assign, no seek, no partition-level offset control. group.type=share is what distinguishes the group at the broker.
What you give up
This part matters more than the feature list.
Ordering. There is none within a share group, not even per key. Cooperative fetching from a partition plus independent redelivery of failed records means record N+1 can complete before record N. If your consumer maintains per-entity state, does read-modify-write, or emits events whose order a downstream reader depends on, a share group is the wrong tool. That is the design, not a tuning problem.
Exactly-once. Share groups are at-least-once. Redelivery after a lock expiry is the normal path, not an exception, and a consumer that crashes after doing its work but before acknowledging will see that record again. Handlers must be idempotent — genuinely, not aspirationally.
Transactional integration. Share-group consumption does not slot into the producer transaction machinery the way a read_committed consume-process-produce pipeline does. If you run exactly-once Streams topologies, share groups are for a different part of your system, not a replacement for that one.
Operational mileage. Per-record state is real state on the broker, kept in an internal share-state topic and cached by the share coordinator. It is bounded by configuration, but it is more broker work per message than "track one offset per partition." At GA in 4.1 this is well-specified and well-tested; it is also, in most estates, less battle-scarred than the consumer path. Plan the first year accordingly.
Where it genuinely fits
The honest use cases are narrow and real:
- Work queues with variable per-item cost. One record takes 50 ms, the next takes 40 seconds. Consumer groups handle this badly because the slow item blocks its partition; a share group lets the other workers keep draining.
- Worker counts that exceed partition counts. You need 200 workers for a burst and the topic has 12 partitions. Repartitioning for peak parallelism stops being a requirement.
- Fan-out to external systems with unpredictable latency. Third-party API calls, where one upstream slowdown currently turns into partition lag.
- Retry semantics you would otherwise hand-roll. Broker-tracked delivery counts and automatic archiving replace a retry topic, a DLQ topic, and the code that shuffles records between them.
Where it does not fit: event sourcing, CDC into a database, materialized views, anything keyed and stateful, anything feeding Kafka Streams. Those are logs and they need log semantics.
If you want to evaluate it
- Get to 4.1 or later. Share groups are GA there. Do not benchmark a 4.0 early-access build and draw conclusions.
- Pick a workload that is already idempotent — not one you plan to make idempotent. The evaluation should test the model, not your ability to retrofit handlers under time pressure.
- Run it beside the existing consumer group. A share group and a consumer group can read the same topic with entirely separate state. Run both on a copy of the traffic and compare throughput, tail latency, and redelivery rate before cutting over.
- Set the lock duration from measured processing time.
group.share.record.lock.duration.msplays the rolemax.poll.interval.msplays for consumers: too short and healthy workers duplicate each other's work, too long and a crashed worker parks records until expiry. Measure p99 handler duration and sit well above it. - Decide the archive policy deliberately.
group.share.delivery.attempt.limitis how many tries a record gets before it drops out of the group's view. Silently archiving a poison record is acceptable only if you alert on the count. - Watch the share coordinator. New broker-side component, new metrics, new failure mode in the runbook. Add the dashboards before the workload matters, not after.
The measured view
Share groups close a real gap. Teams have pushed queue-shaped work through Kafka for a decade using retry topics, partition over-provisioning, and elaborate skip logic, and this replaces a meaningful share of that with mechanics the broker maintains. That is worth something.
It is not a general upgrade to consumer groups, and anyone describing it that way has skipped the ordering paragraph. Most estates will end up running both: consumer groups for the ordered, keyed, stateful flows that are the reason you chose Kafka, and one or two share groups for the job-queue workloads that never fit comfortably. The design work is deciding which of your topics is which — usually a shorter conversation than teams expect, because an ordering requirement either exists or it does not.
If you are weighing share groups against an existing retry-topic scheme, or working out whether a pipeline can tolerate unordered redelivery, tell us about the workload and we will give you a straight answer about which model fits.