MessagePilots

A ZooKeeper-to-KRaft Migration Checklist

ZooKeeper mode is gone in Kafka 4.x: KRaft is the only controller quorum. If you run 3.x with ZooKeeper, the migration is table stakes for staying on supported versions. It is also a genuinely careful operation — the controller quorum is the cluster's brain, and this procedure replaces it while the cluster serves traffic. Here is the checklist we work through with clients, in order.

Before you start

  • Land on a bridge release. Migrate from a recent 3.x (3.6+ is where the migration matured; 3.9 is the last ZooKeeper release). If you are on 2.x, upgrade brokers first — that is its own project.
  • Inventory ZooKeeper dependencies outside Kafka. Old tooling that talks to ZooKeeper directly — legacy kafka-topics --zookeeper scripts, homegrown ACL sync jobs, Burrow-era monitors, anything reading /brokers znodes — breaks the day ZooKeeper goes away. Find these now; the grep is cheaper than the incident.
  • Check feature parity for your setup. Verify your security configuration (SASL mechanisms, authorizer choice) is supported in KRaft mode on your target version. Custom authorizers written against ZooKeeper-era interfaces need porting.
  • Size the controller quorum. Three dedicated controller nodes is the standard answer. Controllers are light on CPU and disk but latency-sensitive — give them their own instances rather than co-locating with loaded brokers, and put them in separate availability zones.
  • Rehearse in staging with a cluster restored from a production metadata snapshot — topic count and ACL volume affect migration time, and a rehearsal is the only place you learn yours.

The migration sequence

  1. Provision the KRaft controllers with a new node.id range that does not collide with broker ids, the migration flag enabled, and ZooKeeper connection details so they can copy metadata.
  2. Enable migration mode on the brokers — restart each broker with the migration flag and controller quorum configuration. The cluster still runs with ZooKeeper as the source of truth.
  3. Wait for metadata migration. The active controller copies topics, configs, ACLs, and producer ids into the KRaft metadata log, then enters dual-write mode: every metadata change is written to both ZooKeeper and KRaft. Watch the controller logs and the ZkMigrationState metric; verify topic and ACL counts match on both sides.
  4. Roll the brokers to KRaft mode — remove ZooKeeper configuration, point them at the controller quorum, restart one at a time with normal health checks between.
  5. Hold the dual-write window. Until you finalize, you can roll brokers back to ZooKeeper mode. Run here for days, not minutes: exercise a deploy, a partition reassignment, a topic creation, an ACL change.
  6. Finalize by taking the controllers out of migration mode. This is the point of no return — after finalization there is no supported path back to ZooKeeper. Decommission ZooKeeper only after finalization, not before.

If you are on MSK

Managed platforms do the controller mechanics for you — on MSK the ZooKeeper-to-KRaft transition arrives as part of a version upgrade rather than a procedure you drive. The pre-flight inventory still applies in full: client versions, IAM/ACL tooling, and anything that ever read ZooKeeper directly (older MSK clusters exposed ZooKeeper endpoints, and someone's script is using them). Managed does not mean inventory-free.

What to watch afterwards

  • Metadata lag: brokers now replicate the metadata log from the controllers; alert on last-applied-offset lag the way you alert on partition lag.
  • Controller quorum health: leader elections, request latency, and disk latency on the controller nodes — a slow controller disk is now a cluster-wide problem.
  • Snapshot behavior: KRaft controllers snapshot the metadata log; confirm snapshots are being produced and cleaned up.

Rollback, honestly

Before finalization, rollback is real but not free: brokers go back to migration mode and then to ZooKeeper mode, and any metadata written only to KRaft during a partial failure needs checking. Practise the rollback in staging too — a rollback nobody has run is a plan, not a capability. After finalization there is no rollback at all, which is why the observation window before it should be measured in days of ordinary operations: a deploy, a reassignment, a topic creation, an ACL change, and ideally one routine broker restart.

The failure modes we plan for

The migration stalls most often on the boring things: a broker missed in the config rollout, an id collision between controllers and brokers, security settings that differ subtly between the ZooKeeper and KRaft paths, and forgotten cron jobs that still expect ZooKeeper. Every one of these is findable in the pre-flight inventory — which is why the inventory, not the restart sequence, is where most of the engineering time belongs.

Treated with that respect, the migration is unglamorous and safe: metadata copies over, brokers roll one at a time, and the scariest step is a config flag you flip after a week of evidence. That is what a good infrastructure migration should feel like.