Poison Pills, Retry Topics, and Dead-Letter Queues: Failing One Record Without Stalling a Partition

A single record that cannot be deserialized will stop a partition forever. The consumer polls it, throws, restarts, polls the same offset, throws again. Lag on that partition climbs in a straight line while every other partition in the topic looks perfectly healthy — which is why the first page is usually "consumer lag high" and the first twenty minutes of the incident are spent looking at brokers that are fine.

Everyone writes a handler for this eventually. The question is what the handler does, and most of the damage we see comes from teams that answered it once, casually, early, and never revisited it. Here is how we think about the three real options — skip, retry, divert — and what each one costs.

First, be precise about what "poison pill" means

The term covers at least four failures with different correct answers:

1. Deserialization failure. The bytes are not valid Avro/Protobuf/JSON, or the schema ID is unknown to this consumer. Retrying never helps; the bytes will not change.
2. Validation failure. The record deserializes but is semantically wrong — a null tenant ID, a negative quantity, a reference to an entity that does not exist. Retrying rarely helps.
3. Transient downstream failure. The database is failing over, the HTTP dependency is returning 503, the rate limiter is engaged. Retrying almost always helps, given time.
4. Bug in the consumer. The record is fine; your code throws on a case you did not anticipate. Retrying helps only after a deploy.

Conflating 1 and 3 is the common mistake. A single catch (Exception e) { retry(); } turns a permanently bad record into an infinite loop, and a single catch (Exception e) { skip(); } silently drops orders during a five-minute database failover. Classify the exception, then act.

Option one: skip, and count it

For deserialization failures specifically, skipping is defensible — the record is unprocessable by definition, and nothing downstream can do better. Two things make it safe rather than reckless:

• Emit a metric per skip, labelled by topic and partition, and alert on a nonzero rate. A silent skip is data loss with extra steps. A counted skip is a known, bounded gap you can go explain.
• Log enough to reconstruct the record: topic, partition, offset, key bytes, headers, and the first few hundred bytes of the payload. "Failed to deserialize record" with no coordinates is useless at 03:00, and the record itself may age out of retention before anyone reads the log.

Kafka Streams has had DeserializationExceptionHandler and ProductionExceptionHandler for years; Connect has errors.tolerance=all. Both default, correctly, to failing fast. Turn tolerance on deliberately and with monitoring attached, not as a way to stop a pager going off.

Option two: retry in place, with a bound

In-place retry is the right answer for transient downstream failure and the wrong answer for everything else. The constraint people forget is max.poll.interval.ms: your retry loop runs inside the poll loop, so if backoff takes longer than that interval the group coordinator evicts the consumer, the group rebalances, and the partition is reassigned mid-retry. You traded one stuck partition for a rebalance that stalls the whole group.

Two mechanisms keep in-place retry honest:

• Bound the total retry time well under max.poll.interval.ms — a few hundred milliseconds of backoff, a handful of attempts, then escalate to diversion. If you genuinely need minutes of backoff, use pause() on the partition and keep calling poll() so the consumer stays in the group, then resume() when the dependency recovers.
• Retry the whole batch, not one record, or be certain your commits are per-offset. The partition is ordered; if record 1001 fails and you skip ahead to 1002, you have made an ordering decision on behalf of every downstream consumer of your output.

Pausing the partition during a sustained outage is usually better than retrying it. Lag grows either way, but a paused consumer is a healthy group member and the lag is honest.

Option three: divert to a retry topic or DLQ

The standard pattern: on failure, produce the record to a side topic with the original key, the original value, and headers recording what happened — exception class, message, source topic, partition, offset, attempt count, timestamp. Then commit the source offset and move on. The partition drains. The failure is preserved.

Kafka Connect has shipped this for a long time (errors.deadletterqueue.topic.name, plus errors.deadletterqueue.context.headers.enable=true, which you want — without it you get the payload and no explanation). Kafka Streams added dead-letter-queue support in its exception handlers in Apache Kafka 4.2, which removes a chunk of the custom plumbing teams have been writing by hand; if you are on 4.2 or later, prefer it to your bespoke version.

Where it gets subtle:

Retry topic versus dead-letter topic are different things. A retry topic is reprocessed automatically, usually by a separate consumer with a delay (retry-5s, retry-1m, retry-10m tiers are a common shape). A dead-letter topic is a parking lot read by humans. Keep them separate. A DLQ that something automatically re-drives is just a slow infinite loop with worse observability.

Diversion breaks ordering, permanently. Record 1001 goes to the retry topic, 1002 is processed immediately, and 1001 lands three minutes later. If these are account balance updates or CDC rows for the same key, you have corrupted state. This is the single thing to decide before adopting the pattern: for key-ordered streams, especially CDC output, the correct behaviour on a non-transient failure is usually to stop that partition, not to let the stream run ahead of a hole.

Schema problems follow the record into the DLQ. If the record failed because its schema is unknown, a DLQ with a typed serializer will fail to write it. Produce DLQ records as raw bytes with a byte[] serializer, and put the diagnosis in headers.

A DLQ nobody reads is a compliance problem. Set retention on it deliberately (long — 30 days or more), alert on records produced to it, and own the replay path. "We'll go look at the DLQ" is not a plan; a documented reprocessing tool that reads the DLQ, filters by error class, and republishes to the source topic is.

Delivery counts and the newer option

Share groups (KIP-932, production-ready as of Kafka 4.2) change the shape of this for queue-style workloads. Each record has a per-consumer acknowledgement and a delivery count, and the broker can move a record aside after a configurable number of failed deliveries. If your workload is genuinely a work queue with no ordering requirement, that is a cleaner home for redelivery semantics than a hand-rolled retry topic. If your workload is an ordered event stream, it is not an option at all — and that distinction, not the feature list, is what should decide it.

What we check in a review

When we run a Kafka health check the error-handling questions are short and the answers are usually revealing:

• What happens to a record your consumer cannot deserialize? Can you name the code path?
• Is there a metric that fires when it happens, and has anyone ever seen it fire?
• Does your retry backoff fit inside max.poll.interval.ms with room for a slow batch?
• Does any topic in this pipeline require per-key ordering? Does your DLQ path preserve it? (These two answers conflict more often than not.)
• Who reads the DLQ, how do they replay from it, and when was that rehearsed?

The goal is not zero bad records. Bad records are a property of a system with more than one team producing to it. The goal is that one bad record costs you a metric and a ticket rather than a partition and an evening.

If you are designing this path now, or cleaning up after a DLQ that quietly collected two million records, tell us what the pipeline carries and we will tell you what we would change first. Engagements are hourly, and the output is written findings you can act on without us.