Running Kafka on Kubernetes used to be a contrarian choice. It isn't any more: with KRaft removing ZooKeeper from the picture and the Strimzi operator handling most of the lifecycle, a Kubernetes-hosted cluster is a reasonable default for teams that already run everything else there. What the operator removes is toil, not decisions. The decisions that remain are the ones that decide whether your cluster is boring.
This is a field-notes list of the ones we end up discussing on nearly every engagement.
Node pools are the unit you should be thinking in
Strimzi's KafkaNodePool resource lets a single logical Kafka cluster be composed of several groups of nodes, each with its own roles (controller, broker, or both), storage, and resources. With KRaft this is no longer optional detail — it is how you express cluster topology.
Two practical consequences:
- Separate controllers from brokers on anything carrying real traffic. Combined-role nodes are fine for development and awkward in production: controller quorum health should not depend on a node that is also serving a 400 MB/s fetch workload, and the two have very different resource shapes. Three dedicated controllers with modest CPU and fast, small disks is the usual answer.
- Node pools make heterogeneous rollouts possible. Moving brokers to a new instance type, a new storage class, or a new availability zone becomes "add a pool, reassign partitions onto it, scale the old pool down" rather than a mutation of every broker in place. That is the same shape as a migration you would do outside Kubernetes, and it is equally reversible.
If you are still on a ZooKeeper-based Strimzi cluster, the operator's migration path moves you through node pools anyway; treat that as an opportunity to get the topology right rather than a mechanical lift.
Storage: local disks are fast and local disks are a commitment
Kafka's performance story assumes the page cache and sequential writes to attached disk. On Kubernetes you choose between network-attached volumes (EBS gp3, GCP pd-balanced) and local NVMe exposed as local PersistentVolumes.
Network-attached is the right default for most teams. It survives node replacement: the pod reschedules, the PVC follows, and the broker restarts with its log directory intact — no re-replication of terabytes. The cost is IOPS and throughput limits you must provision explicitly and watch. A gp3 volume at default settings will cap a busy broker long before the CPU does.
Local NVMe is meaningfully faster and meaningfully more work. Losing a node means losing a replica's data and re-replicating from scratch, which must fit inside your replication throughput budget and your min.insync.replicas margin. Choose it when measured disk latency is actually your bottleneck, not on principle.
Two settings that save grief either way: set storageClass with allowVolumeExpansion: true so you can grow a log directory without a rebuild, and never let the operator's deleteClaim: true near production. Reclaiming a PVC on scale-down is not something you want to discover by accident.
Rack awareness has to be wired to real zones
Set rack.topologyKey: topology.kubernetes.io/zone on the Kafka resource. Strimzi then sets each broker's broker.rack from its node's zone label, and the replica placement logic spreads partition replicas across zones. Without it, a replication factor of 3 can place all three replicas in one availability zone, and a zone outage takes the partition offline despite the arithmetic looking healthy.
Pair it with topology spread constraints on the broker pods themselves, so the scheduler distributes pods across zones rather than packing them. Rack awareness tells Kafka where replicas should go; spread constraints tell Kubernetes where pods go. Both are needed, and only one of them is on by default in anybody's config.
Rack awareness also unlocks client.rack and follower fetching, which is how you stop paying for cross-AZ reads — worth doing for the bill as much as for the availability.
Rolling updates: the operator is careful, your settings may not be
Strimzi rolls brokers one at a time and waits for the restarted broker's partitions to come back in sync before continuing. That is the right behaviour, and it is also why a rolling update of a large cluster takes hours rather than minutes. Plan for that; don't interrupt it halfway.
The parts that bite:
- Under-replicated partitions stall the roll, correctly. If a topic has replication factor 1, or
min.insync.replicasequal to the replication factor, the operator cannot restart a broker without taking partitions below the threshold. Audit topic configs before an upgrade, not during one. - PodDisruptionBudgets protect against Kubernetes, not against you. Strimzi manages a PDB with
maxUnavailable: 1. A node drain during an operator-driven roll can still produce two brokers down at once if you drain impatiently. Coordinate cluster maintenance windows with the operator's work. - Readiness probes are not sync checks. A broker pod reporting ready has an open listener; it may still be catching up on replication. Watch
UnderReplicatedPartitionsacross the roll, not pod status. - Change batching. A config change that requires a restart and an image bump applied together cost one roll; applied separately they cost two. Batch deliberately.
What we would check before calling it production
- Dedicated controller node pool, three nodes, separate storage
allowVolumeExpansionon the storage class;deleteClaim: false- Rack awareness from the real zone label, plus pod topology spread
- Every production topic at replication factor 3 with
min.insync.replicas=2 - Resource requests equal to limits for brokers, so they land in the Guaranteed QoS class and are not evicted under node pressure
- JVM heap modest (6–8 GB is usually plenty) with the rest of the pod's memory left to the page cache
- Alerting on under-replicated partitions, offline partitions, and controller quorum state — not just pod restarts
The operator makes a Kafka cluster declarative. It does not make the trade-offs go away, and it will faithfully roll out a topology that loses data in a zone failure if that is what the YAML says. Read the YAML as a cluster design, because that is what it is.
If you are planning a move onto Kubernetes, or running a Strimzi cluster that feels fragile around upgrades, a Kafka health check covers exactly this ground: topology, replica placement, storage sizing, and upgrade posture, with findings ranked by production risk.