October 19, 2026
One unparseable record can stall a partition indefinitely. The three honest options — skip, retry, divert — what each costs, and why a DLQ quietly breaks per-key ordering.
Read moreOctober 12, 2026 · MessagePilots
Both will do your joins and aggregations. They differ in who owns state, how you recover, and what the on-call rotation looks like. A decision framework, plus the cases where neither is the answer.
Read moreSeptember 21, 2026
Connect sink, Flink writer, or a managed materialization like Tableflow: the three ways a Kafka topic becomes an Iceberg table, and the commit, small-file, and schema questions that decide which one survives contact with production.
Read moreSeptember 21, 2026
Writing partitions straight to S3 removes cross-AZ replication and makes brokers stateless, at the cost of hundreds of milliseconds of latency. Which topics that trade suits, what the request bills and the batch coordinator add back, and how to test it on one topic.
Read moreSeptember 16, 2026
Stateful Kafka Streams applications fail in predictable ways: off-heap RocksDB memory, eleven-minute restores, changelog topics nobody sized. The settings and habits that keep them boring.
Read moreSeptember 15, 2026 · MessagePilots
Connect is easy to start and easy to run badly. Field notes on tasks.max, the rebalance protocol, internal topics, error handling and DLQs, converters, and what exactly-once source support really requires.
Read moreSeptember 14, 2026 · MessagePilots
Kafka 4.1 made share groups generally available: many consumers per partition, per-record acknowledgement, broker-tracked redelivery. What that buys job-queue workloads, what it costs in ordering and exactly-once, and how to evaluate it.
Read moreSeptember 11, 2026 · MessagePilots
Snapshots that stall, replication slots that fill a disk, tombstones that vanish under the wrong cleanup policy. The change-data-capture decisions worth making before the connector goes to production.
Read moreAugust 10, 2026
EOS delivers exactly-once processing inside a Kafka-to-Kafka pipeline — not exactly-once side effects. Where transactions earn their latency cost, and where an idempotent sink is the better buy.
Read moreJuly 13, 2026
Partition count caps your parallelism and is painful to change on a keyed topic. A worksheet method: measure per-partition throughput, size for peak plus replay headroom, check key skew.
Read moreJune 16, 2026
Compatibility modes decide your deploy order, transitive checks protect you from old data, and the rest is a short list of rules a CI gate can enforce. How to change schemas without incidents.
Read more