October 26, 2026 · MessagePilots
An operator removes most of the toil of running Kafka on Kubernetes, but not the decisions. Storage classes, rack awareness, KRaft node pools, and rolling-update behaviour still belong to you.
Read moreOctober 19, 2026
One unparseable record can stall a partition indefinitely. The three honest options — skip, retry, divert — what each costs, and why a DLQ quietly breaks per-key ordering.
Read moreOctober 12, 2026 · MessagePilots
Both will do your joins and aggregations. They differ in who owns state, how you recover, and what the on-call rotation looks like. A decision framework, plus the cases where neither is the answer.
Read moreOctober 5, 2026 · MessagePilots
Replication factor 3 does not mean your writes survive. The producer and broker settings that decide whether an acknowledged record is still there after a broker dies, and the availability you trade for them.
Read moreSeptember 28, 2026 · MessagePilots
One shared cluster, twenty teams, and no client quotas is a latency incident waiting for its trigger. How broker-side quotas, namespacing, and per-tenant visibility keep a shared Kafka estate boring.
Read moreSeptember 21, 2026
Kafka never moves partitions on its own. How to plan a reassignment in byte-sized batches, size the replication throttle from measured headroom, decide when Cruise Control earns its keep, and drain brokers safely on the way out.
Read moreSeptember 21, 2026
Writing partitions straight to S3 removes cross-AZ replication and makes brokers stateless, at the cost of hundreds of milliseconds of latency. Which topics that trade suits, what the request bills and the batch coordinator add back, and how to test it on one topic.
Read moreSeptember 18, 2026 · MessagePilots
Turning on authentication and authorization is the change most likely to break every client at once. The listener model, the KRaft-era authorizer, and a rollout order that keeps traffic flowing while you tighten it.
Read moreSeptember 17, 2026 · MessagePilots
On a multi-AZ cluster, inter-zone data transfer often costs more than the brokers. Rack awareness, fetch-from-follower, producer partitioning, and an honest look at what diskless designs change.
Read moreSeptember 16, 2026
Stateful Kafka Streams applications fail in predictable ways: off-heap RocksDB memory, eleven-minute restores, changelog topics nobody sized. The settings and habits that keep them boring.
Read moreSeptember 15, 2026 · MessagePilots
Connect is easy to start and easy to run badly. Field notes on tasks.max, the rebalance protocol, internal topics, error handling and DLQs, converters, and what exactly-once source support really requires.
Read moreSeptember 14, 2026 · MessagePilots
Replicating a Kafka cluster is easy to start and hard to fail over to. What MirrorMaker 2 copies, what offset translation can and cannot promise, how active/active avoids loops, and the failover rehearsal that tells you whether any of it works.
Read moreSeptember 14, 2026 · MessagePilots
Kafka 4.x ships a broker-side rebalance protocol that removes the global sync barrier and moves timeouts into group configs. What actually changes for your consumers, and a migration order that does not need a maintenance window.
Read moreSeptember 14, 2026 · MessagePilots
Kafka 4.1 made share groups generally available: many consumers per partition, per-record acknowledgement, broker-tracked redelivery. What that buys job-queue workloads, what it costs in ordering and exactly-once, and how to evaluate it.
Read moreSeptember 10, 2026
Tiered storage (KIP-405) moves closed log segments to object storage and shrinks broker disks. What it changes about retention, rebalancing, and read latency — and the topics it still will not accept.
Read moreJuly 13, 2026
Partition count caps your parallelism and is painful to change on a keyed topic. A worksheet method: measure per-partition throughput, size for peak plus replay headroom, check key skew.
Read moreMay 19, 2026
KRaft is the only controller quorum in Kafka 4.x. A field checklist for the migration: pre-flight inventory, the dual-write sequence, the finalization decision, and what to monitor after.
Read moreApril 14, 2026
Rebalance storms, auto-commit data loss, lag measured in the wrong unit: the recurring consumer-group mistakes behind most Kafka incidents we see, and the specific fixes.
Read more