Change data capture is the most common way Kafka gets its first serious workload. Someone needs the orders table in a search index, Debezium reads the database log, and within a quarter half the company is consuming those topics. The connector itself is a few dozen lines of JSON. The decisions around it are what determine whether the pipeline is boring a year later.
These are the ones we keep making on client engagements, roughly in the order they bite.
The snapshot is the hard part, not the streaming
Steady-state CDC is cheap: the connector tails the transaction log and emits a few thousand events a second without noticing. The initial snapshot is where projects stall. Debezium's default initial mode reads every selected table in full before it starts streaming, and on a multi-terabyte table that can run for hours or days while the log position it must eventually resume from keeps advancing.
Three things follow from that:
- Size the snapshot before you schedule it. Row count multiplied by average serialized row size is the volume you are about to write into Kafka, and it usually dwarfs a day of change traffic. Retention and disk on the target cluster need to absorb it.
- Prefer incremental snapshots (
incremental, via the signalling table) on large tables. They run in chunks, interleaved with live streaming, so the connector never goes dark and a failure costs you one chunk rather than the whole pass. They are also the only sane way to add a table to an existing connector later. - Plan for a re-snapshot. You will need one — a new column backfilled, a corrupted topic, a consumer that needs history it never kept. Knowing in advance whether that means "signal an incremental snapshot" or "take the pipeline down for two days" is the difference between a routine task and an incident.
The replication slot is a disk-space time bomb
On PostgreSQL, the connector holds a logical replication slot, and the database cannot recycle WAL segments newer than that slot's confirmed position. Stop the connector on a Friday, leave the slot in place, and a busy database can fill its volume over the weekend. The failure lands on the database, not on Kafka, which is why it surprises people.
SQL Server and MySQL have their own versions of the same dependency (capture-table retention, binlog expiry): the connector is pinned to a window of log history, and if it falls outside that window it cannot resume — it can only re-snapshot.
What to do about it:
- Alert on replication-slot lag in bytes, with a threshold that gives you hours of headroom, and route it to whoever owns the database.
- Alert on connector task state directly. A
FAILEDtask is silent otherwise; Connect will happily sit there with a dead task and a live slot. - On a low-traffic database, configure a heartbeat (
heartbeat.interval.msplus a heartbeat action query) so the slot's position keeps advancing even when the captured tables are idle. Otherwise a quiet table on a busy server pins WAL indefinitely. - Write down the decision about what happens to the slot when the connector is deliberately stopped for a long window. "Drop it and re-snapshot on return" is a legitimate answer; "nobody decided" is not.
Key the topic the way the data is actually ordered
Kafka guarantees order per partition, and Debezium keys each record by the table's primary key, so all changes to one row land on one partition in commit order. That is the guarantee most consumers silently depend on.
It breaks in predictable ways. Rewriting the key with a Single Message Transform to something coarser or finer than the primary key changes which changes are ordered relative to each other. Increasing a topic's partition count re-hashes keys, so records for one row can sit in two partitions with no order between them — the same trap as any keyed topic, but harder to notice because the pipeline keeps working for rows that never changed during the cutover. And ordering is per-row only: if a consumer needs a parent and child row change applied together, CDC will not hand it to them atomically. That is what the outbox pattern is for — the application writes one event row describing the business fact, the connector streams that table, and the ordering problem disappears into the transaction that produced it.
Deletes, tombstones, and compaction have to agree
A delete in Debezium is two records: an envelope with op: d carrying the before-image, then a tombstone — same key, null value — which is what log compaction uses to actually drop the key.
This only works if the pieces line up:
- If the topic is compacted, keep
tombstones.on.deleteenabled, or deleted rows live in the topic forever and any consumer rebuilding state from the beginning resurrects them. - If the topic is delete-retention only, the tombstone is just another record — fine, but your consumer must handle a null value without throwing. Deserializers that assume a non-null payload are a classic poison pill on the first production delete.
delete.retention.msbounds how long tombstones survive on a compacted topic. A consumer that is offline longer than that can miss the delete entirely and keep a row that no longer exists upstream. If a downstream store must be exactly consistent, make sure the bootstrap window is well inside that setting.
Schema changes arrive whether or not you are ready
DDL upstream shows up as a new schema on the topic. With Schema Registry and BACKWARD compatibility, adding a nullable column is a non-event. Dropping a column or narrowing a type is not, and the registry will reject it — which is the correct outcome, but it means a database migration can now fail a connector, and the database team probably does not know that.
The practical version of this is a short contract: captured tables are additive-only unless a change is coordinated with the consumers; the compatibility mode is set per subject and enforced in CI; someone has stated who gets told before a migration touches a captured table. A SET NULL on a schema field is a much cheaper conversation than an emergency re-snapshot.
Connect is a cluster too
Kafka Connect fails in its own ways, separately from the brokers. The three that account for most of our CDC incident calls:
- A single connector monopolising a worker. CDC source connectors are single-task by design — one task per database, reading one log. A large snapshot on one connector can starve everything else sharing the worker. Isolate high-volume CDC on its own Connect cluster if it matters.
- Offsets living in a place nobody backs up. Connect stores source offsets (the log position) in an internal compacted topic. Lose it, and every connector re-snapshots. It deserves the same replication factor and care as any production topic, and it needs to be part of your disaster-recovery plan.
- Converter configuration drift. Key and value converters are set per-worker and overridable per-connector. Two connectors writing what looks like the same format, one with schemas enabled and one without, produce topics that only some consumers can read.
What to monitor
Four signals cover most of it:
- Connector and task state — anything not
RUNNING, alert immediately. - Replication-slot / binlog lag in bytes or seconds, owned jointly with the database team.
MilliSecondsBehindSourcefrom the Debezium metrics — how stale the stream is relative to the database, which is the number your downstream SLO is really about.- Snapshot progress during any snapshot, so a stalled one is visible before the WAL becomes the story.
None of this is exotic. It is the same discipline as the rest of a Kafka estate: know your retention, know your ordering guarantee, know which component fails first and who gets paged. CDC just adds a second system — the database — that shares the blast radius.
If you are standing up CDC into Kafka, or you have a pipeline that snapshots fine and then drifts, our Streaming Architecture & Build and Operations & Capacity work covers exactly these decisions. Tell us about the pipeline and we will tell you what we would check first.