Field notes from running Kafka in production.
September 21, 2026
Connect sink, Flink writer, or a managed materialization like Tableflow: the three ways a Kafka topic becomes an Iceberg table, and the commit, small-file, and schema questions that decide which one survives contact with production.
Read moreSeptember 21, 2026
Kafka never moves partitions on its own. How to plan a reassignment in byte-sized batches, size the replication throttle from measured headroom, decide when Cruise Control earns its keep, and drain brokers safely on the way out.
Read moreSeptember 21, 2026
Writing partitions straight to S3 removes cross-AZ replication and makes brokers stateless, at the cost of hundreds of milliseconds of latency. Which topics that trade suits, what the request bills and the batch coordinator add back, and how to test it on one topic.
Read moreSeptember 18, 2026 · MessagePilots
Turning on authentication and authorization is the change most likely to break every client at once. The listener model, the KRaft-era authorizer, and a rollout order that keeps traffic flowing while you tighten it.
Read moreSeptember 17, 2026 · MessagePilots
On a multi-AZ cluster, inter-zone data transfer often costs more than the brokers. Rack awareness, fetch-from-follower, producer partitioning, and an honest look at what diskless designs change.
Read more