Rebalancing Brokers Without an Incident: Reassignment, Throttles, and Cruise Control

Adding brokers is the easy part. The cluster accepts them, they show up in the metadata, and then they sit there holding no data while the original brokers stay pinned at 85% disk. Kafka does not move partitions on its own. Reassignment is a manual, throttled, deliberately boring operation — and the teams that get paged during it are almost always the ones who ran it with no throttle and no plan for what happens if it stalls.

This is roughly what we do on engagements when a cluster needs to grow, shrink, or even out.

What a reassignment actually does

kafka-reassign-partitions.sh writes a desired replica assignment into cluster metadata. Each new replica then starts as a follower outside the ISR: it fetches the partition from the leader, from the earliest retained offset forward, until it catches up and joins the ISR. Only then is the old replica dropped.

Three consequences follow, and they explain nearly every reassignment incident:

1. You are replaying retention over the network. Moving one replica of a 500 GB partition means transferring 500 GB. A "small" rebalance of 200 partitions can be tens of terabytes of inter-broker traffic. On cloud infrastructure it is also cross-AZ traffic you pay for.
2. The source broker does the reading. Fetches from catching-up followers compete with production reads for page cache and disk. A leader that was comfortably serving from cache starts hitting disk, and produce latency rises on a broker nobody changed.
3. Under-replicated partitions are expected, temporarily. During a move, UnderReplicatedPartitions goes non-zero by design. If that metric is your primary alert, you will either page yourself all afternoon or learn to ignore the alert that matters.

Throttle first, then move

The throttle is not optional. Set it before the reassignment starts, not after you notice latency:

kafka-reassign-partitions.sh --bootstrap-server broker:9092 \
  --reassignment-json-file plan.json --execute \
  --throttle 50000000

That is bytes per second per broker (50 MB/s here), applied to leader and follower replication for the throttled replicas. Pick it from measured headroom rather than a round number you like: take the broker's sustained network and disk read throughput, subtract peak production load plus the margin you want for a traffic spike, and throttle to what remains. Half of spare capacity is a reasonable starting point.

Two details that catch people:

• The throttle is stored as dynamic broker config (leader.replication.throttled.rate / follower.replication.throttled.rate) plus per-topic replica lists. If the reassignment is cancelled or the tooling dies mid-run, the throttle can be left behind, silently capping replication for a later, unrelated failure recovery. Always finish with --verify, which clears it, and confirm with kafka-configs.sh --describe --entity-type brokers.
• You can raise the throttle mid-flight by re-running --execute with the same plan and a higher value. Start conservative and increase; that is much cheaper than starting aggressive and explaining the latency graph afterwards.

Sizing the batch

Do not submit one plan containing every partition. Kafka starts all of the moves at once, and the throttle is per broker rather than per move, so you lose the ability to reason about completion time or to back out cleanly.

Batch by bytes, not partition count. Take the largest partitions first — they dominate the transfer and they are the ones that blow the estimate — and build plans of a few hundred gigabytes each. At 50 MB/s a 300 GB batch takes roughly two hours: a unit of work you can run, verify, and stop between. Ten of those over two days is a rebalance nobody notices. One 3 TB submission is an afternoon of "is it still going?"

Reducing retention temporarily on the biggest topics before a move is also legitimate. If a topic keeps 14 days and the business genuinely needs 3, you have just cut the transfer by three quarters.

Cruise Control, and when it earns its keep

Hand-written plans stop scaling somewhere around a few hundred partitions and one dimension of balance. Cruise Control (LinkedIn's, open source, works against KRaft clusters through the admin API) models the cluster as a set of goals — replica counts, disk usage, leader distribution, inbound and outbound network per broker, rack awareness — and produces a proposal that satisfies hard goals while optimizing soft ones.

What makes it worth running:

• It balances on network and disk load, not just partition count. An even partition count across brokers is not balance if three of those partitions carry most of your throughput.
• It has add_broker and remove_broker workflows, so scaling becomes an API call plus a review rather than a hand-built JSON file.
• Self-healing can respond to a dead broker automatically. We usually suggest leaving that off at first and running proposals in dry-run for a few weeks until the team trusts the goals it is optimizing.

What to watch for: Cruise Control needs its metrics reporter deployed on the brokers and a sampling window — typically hours to a day — before proposals mean anything. Its concurrency settings (num.concurrent.partition.movements.per.broker) are your throttle equivalent and default to values that are optimistic for large partitions. And it will happily propose a perfectly balanced cluster that requires moving 40 TB, so read the data-movement estimate in the proposal before approving. Every time.

If you run Kafka on Kubernetes, Strimzi exposes Cruise Control through KafkaRebalance resources, which is a reasonable middle ground: declarative proposals, human approval, no bespoke tooling to maintain.

The cheap move: leader rebalancing

Before you move any data, check whether the problem is leadership rather than placement. Leaders serve all produce and consume traffic for a partition; followers only replicate. After a broker restart, leadership does not necessarily return to the preferred replica promptly — auto.leader.rebalance.enable is on by default, but the check interval and imbalance threshold mean it can lag.

kafka-leader-election.sh --election-type preferred --all-topic-partitions moves leadership back with no data transfer at all. If one broker is hot on network while its disk usage matches its peers, this is a 30-second fix, and it is the first thing to try.

Shrinking a cluster

Removing brokers is the same operation in reverse, with one addition: the departing brokers must be fully drained of replicas before shutdown, and you must confirm it. Generate a plan that assigns them zero partitions, run it in batches, verify, then check kafka-log-dirs.sh on those brokers for empty log directories. Shutting down a broker that still leads a min.insync.replicas=2 partition with only two replicas remaining makes that partition immediately unwritable for producers.

Rack awareness deserves a second look here. If your plan pulls the last replica of a partition out of an availability zone, you have quietly turned a three-AZ topic into a two-AZ one, and nothing will tell you until the zone you depend on has a bad day.

Tiered storage changes the arithmetic

On a cluster with tiered storage enabled, only local segments move during a reassignment — remote segments stay in object storage and the new replica simply learns about them. A 500 GB partition with 480 GB tiered is a 20 GB transfer.

That is the strongest operational argument for tiered storage and it is underweighted in most evaluations: it makes broker replacement, scaling, and recovery fast again. If your rebalances currently take days, that is worth pricing.

A runbook that holds up

1. State the goal as a metric: disk skew, network skew, or a broker being added or removed. "It looks uneven" is not a goal.
2. Try preferred-leader election first. Re-measure.
3. Capture a baseline: per-broker network in/out, disk read throughput, produce p99, consumer time lag.
4. Build the plan in byte-sized batches, largest partitions first.
5. Set a throttle from measured headroom. Write down the number and the reasoning.
6. Execute one batch. Watch produce p99 and follower fetch rate, not only partition progress.
7. Run --verify to completion, confirm the throttle config is gone, re-measure.
8. Next batch. Stop at the end of the day with the cluster in a verified state, not mid-move.

None of this is clever. The judgment is in deciding it does not need to be: a rebalance that takes three unremarkable days is better than one that takes four hours and an incident review.

If a cluster needs to grow, shrink, or even out and you would rather rehearse it with someone who has done it on a live estate, Operations & Capacity is that engagement — measured plan, throttle sizing, and a written runbook your team can run again next quarter. Tell us about the cluster.