"We need exactly-once" appears in most streaming requirements documents. It is worth being precise about what Kafka's exactly-once semantics (EOS) actually deliver, because the honest version — exactly-once within a Kafka-to-Kafka pipeline — is narrower than the phrase suggests, and both over-buying and under-buying it are expensive.
What EOS is
Three mechanisms stack up:
Idempotent producers (default since Kafka 3.0) stamp each batch with a producer id and sequence number, so broker-side retries cannot write duplicates within a partition. This is free — take it everywhere.
Transactions let a producer write to several partitions atomically: all the writes become visible together or not at all. Consumers set isolation.level=read_committed to see only committed data.
Consume-process-produce atomicity is the part people actually mean by EOS: a transaction that bundles the output writes and the input offset commits into one atomic unit. Crash mid-way and neither the outputs nor the offset advance — the retry reprocesses the input and rewrites the output, but only one committed copy is ever visible. Kafka Streams packages this whole pattern as processing.guarantee=exactly_once_v2; hand-rolled clients use sendOffsetsToTransaction. Zombie fencing (via transactional.id) keeps a paused-and-resumed old instance from committing stale work behind its replacement.
What EOS is not
The transaction boundary is Kafka. The moment your processor touches the outside world — calls a payment API, writes a database row, sends an email — that side effect is not in the transaction. Crash after the API call but before the commit and the retry calls the API again. Exactly-once processing does not give you exactly-once side effects.
For external effects you still need one of the classic patterns: an idempotent sink (deduplicate on a business key; the database upsert is your friend), a transactional outbox on the database side, or an end-to-end idempotency key carried through the pipeline. Which means: if your pipeline's terminal step is an idempotent database write anyway, at-least-once delivery plus that idempotent sink already gives you effectively-once results — without any of the costs below.
What EOS costs
- Latency. Committed data becomes visible at transaction commit, so end-to-end latency is floored by the commit interval (Streams commits every 100 ms by default under EOS). Sub-100 ms pipelines and EOS are in tension.
- Throughput. Transaction begin/commit round-trips and coordinator writes add overhead — much better since v2 pooled things per-instance rather than per-task, but not zero, and small transactions amplify it.
- Operational surface. A new failure vocabulary:
ProducerFencedException,InvalidTransactionTimeoutException, transaction coordinator load, transactions abandoned by crashed producers blockingread_committedconsumers until timeout. Your runbooks grow. - Everyone must play. One
read_uncommittedconsumer or one non-transactional producer in the pipeline and the guarantee quietly isn't.
Where teams get it wrong in both directions
Bought and unused. A pipeline runs exactly_once_v2 end to end and finishes by POSTing to an external service. The transaction covers the Kafka writes; the POST happens regardless. The team believes duplicates are impossible, so nobody built the deduplication that would actually prevent them — the guarantee bought the wrong thing, and paid latency for it.
Skipped and needed. A Streams topology maintains running aggregates — counts, sums, sessionized state — under at-least-once. Reprocessing after a crash double-counts into the state store, and the drift is silent: no error, no alert, just numbers that stop matching the source system. When state is derived from a stream, at-least-once is not "slightly imprecise"; it is permanently wrong in a way nothing reconciles.
Half-enabled. One consumer left at read_uncommitted (the default) sees aborted transaction data. One legacy producer writing to a topic in the chain non-transactionally. The pipeline diagram says exactly-once; the wire does not.
Auditing for these takes an afternoon: list every producer, consumer, and connector on each topic in the pipeline, and record its isolation level and transactional status. The gaps are usually obvious once written down — and the audit is the deliverable, not the config change.
How to decide
Ask two questions of each pipeline:
- Where do results leave Kafka? If the terminal sink is idempotent or transactional on its own side, prefer at-least-once + idempotent sink — simpler, faster, and the guarantee ends up equivalent where it matters.
- Is duplicate processing itself harmful? Multi-stage Streams topologies with aggregations are the strong EOS case: reprocessing corrupts counts and joins, the pipeline is Kafka-to-Kafka, and
exactly_once_v2is one configuration line. Turn it on and budget the commit-interval latency.
The pattern we see repeatedly: EOS inside the streaming core where reprocessing would corrupt state, at-least-once with idempotency at the edges where the world is being changed. Teams that articulate the boundary this way stop debating "exactly-once: yes or no" and start writing down, per topic, which guarantee holds and why — which is the document your next incident review will wish existed.