Running Kafka Connect in Production: The Parts That Bite

Kafka Connect is the least glamorous part of most streaming estates and, in our experience, the part most likely to be running on defaults nobody revisited. A connector is a few lines of JSON, so it rarely gets the design review a Streams application gets — and then a year later it is the thing carrying your CDC feed into the warehouse.

These are the areas where we consistently find problems during a health check, and what we change.

1. tasks.max is a ceiling, not a request

Connect splits a connector into tasks and distributes them across workers. tasks.max sets an upper bound; the connector decides how many tasks it can actually use, and the limit is usually structural:

• A sink connector can use at most one task per partition of its input topics. tasks.max=16 on a 4-partition topic gives you 4 tasks and 12 units of wishful thinking.
• A JDBC source in table mode is bounded by the number of tables.
• A Debezium source for most databases is a single task by design, because it reads one ordered replication stream. No amount of tasks.max changes that; if a Debezium connector is your bottleneck, the fix is downstream parallelism or splitting connectors by table set, not a bigger number.

Check the running task count against the configured maximum (GET /connectors/{name}/status) rather than trusting the config. A connector quietly running one task while the config says twelve is a common cause of "we scaled the Connect cluster and nothing got faster."

2. Rebalancing: make sure you are on the incremental protocol

Older Connect versions stopped every task on every worker whenever one connector was added, reconfigured, or a worker left the group. On a cluster hosting thirty connectors, a single config change paused everything for the duration of the rebalance.

Modern versions default to incremental cooperative rebalancing, which moves only the affected tasks. Two things to verify:

• connect.protocol=sessioned (or at minimum compatible) on every worker — mixed settings fall back to the weakest.
• scheduled.rebalance.max.delay.ms (default 5 minutes) is tuned for your deploy pattern. It is the grace period Connect waits before reassigning the tasks of a departed worker, so a rolling restart does not shuffle work twice. If your workers restart in 40 seconds, five minutes of idle tasks may be worse than a reassignment; if they take eight minutes, raise it.

3. The internal topics decide whether the cluster survives

A distributed Connect cluster keeps its state in three topics: configs, offsets, and status. We still find these created with defaults from a laptop demo.

• config.storage.topic: one partition, compacted, replication.factor=3, min.insync.replicas=2. More than one partition is a corrupted cluster waiting to be discovered; ordering matters.
• offset.storage.topic: compacted, replicated, typically 25 partitions. Losing it means source connectors restart from their configured beginning — for a CDC connector, that can mean a full re-snapshot.
• status.storage.topic: compacted, replicated, 5 partitions.

Also keep group.id and all three topic names unique per Connect cluster. Two clusters sharing a group.id will fight over task assignment in ways that look like random connector failures. Back up connector configs outside Kafka (the REST API's config endpoints are enough) so a bad broker day is not also a lost pipeline inventory.

4. Error handling: decide what "bad record" means before it arrives

By default a sink connector fails the task on the first record it cannot convert or write. The task goes to FAILED, stops consuming, and lag grows until someone notices — and the noticing is the problem, because a failed task does not necessarily mean a failed connector in most dashboards.

The knobs:

errors.tolerance=all
errors.deadletterqueue.topic.name=dlq.<connector>
errors.deadletterqueue.context.headers.enable=true
errors.deadletterqueue.topic.replication.factor=3
errors.log.enable=true
errors.log.include.messages=false

Two cautions. First, errors.tolerance=all without a DLQ is silent data loss; the pair is not optional. Second, DLQs only catch conversion, transformation, and delivery errors — not everything a connector can do wrong. And a DLQ with no alert on its message rate is a topic where bad records go to be forgotten. Set errors.log.include.messages=false if the payloads carry regulated data; headers alone usually identify the offset well enough to investigate.

5. Converters are a contract, not a serialization detail

The converter is where Connect meets your schema strategy. Three recurring mistakes:

• JsonConverter with schemas.enable=true embeds a schema in every record, often tripling message size on high-volume topics. If you want schemas, use Avro or Protobuf with Schema Registry; if you want small JSON, disable embedded schemas and accept that downstream typing is your problem.
• Key and value converters are configured separately, and a StringConverter key with an Avro value is a perfectly reasonable choice — but it must be stated, not inherited from the worker defaults by accident.
• Worker-level converter defaults make connectors non-portable between clusters. Set converters per connector.

Getting this wrong is how a sink starts writing null columns after a producer change, and the topic itself looks healthy the whole time.

6. Exactly-once source support is real, and it has prerequisites

Connect gained exactly-once support for source connectors (KIP-618). It works, but it is not a single flag:

• Every worker in the cluster needs exactly.once.source.support=enabled, rolled out in the documented two-phase sequence (preparing on all workers first, then enabled).
• Workers need transactional producer permissions, including Write and IdempotentWrite on the offsets topic and the relevant transactional IDs.
• The source connector must implement the required hooks; not all do.

Sink connectors are a different story: exactly-once there depends on the destination system supporting idempotent or transactional writes, and consumer.override.isolation.level=read_committed if you are reading transactional topics. If the sink cannot deduplicate, you have at-least-once with extra steps — which is often fine, but should be written down rather than assumed.

7. Monitor tasks, not connectors

The minimum set we like to see alerting on:

• Task state per connector: anything in FAILED, and anything that has been UNASSIGNED for more than a rebalance delay.
• Sink lag per consumer group (Connect sinks are ordinary consumer groups — connect-<connector-name> — so your existing lag tooling already works, in seconds rather than records).
• Source poll-batch-avg-time-ms and produce error rate.
• DLQ topic message rate, per connector.
• Worker rebalance rate — a cluster rebalancing continuously is usually one crash-looping task away from being understood.

And restart failed tasks with intent. POST /connectors/{name}/restart?includeTasks=true&onlyFailed=true is the useful form; a blanket connector restart during an incident tends to add a rebalance to whatever you were already dealing with.

The short version

Connect earns its place precisely because it is boring: no application code to own, no bespoke consumer to tune. But the defaults are tuned for getting started, not for carrying production data flows. Task counts bounded by structure, internal topics replicated and single-partitioned where it matters, an explicit bad-record policy with a DLQ that someone watches, converters stated per connector, and alerting at the task level cover most of what goes wrong.

If you are running a Connect cluster whose failure modes you have not yet had to explain to anyone, a review is cheaper before the first incident than after it. Our Kafka health check covers Connect alongside the brokers and consumer groups, and Streaming Architecture & Build covers pipeline design when the answer is that the connector is the wrong tool for the job. Get in touch with the specifics of your pipeline and we will tell you where we would start.