Cross-AZ Traffic: The Line Item That Dominates Cloud Kafka Bills

Most cost conversations about Kafka in the cloud start with instance types and end with storage. Then somebody opens the networking section of the bill. On a three-AZ cluster with replication factor 3, inter-zone data transfer is frequently the largest single line item — sometimes larger than the brokers carrying the traffic. It is also the line item teams understand least, because nothing in the Kafka configuration file is named "cost."

This is a walk through where the bytes actually cross a zone boundary, which of those crossings you can remove without weakening durability, and which ones are the price of the availability you asked for.

Count the zone crossings for one record

Take a single 1 KB record produced to a topic with RF=3 across three availability zones, consumed by two consumer groups. Assume nothing is rack-aware.

1. Produce. The client connects to the leader of the target partition, wherever it lives. With brokers spread evenly over three zones, roughly two-thirds of produce traffic leaves the producer's zone. → ~0.67 KB cross-AZ.
2. Replication. The leader ships the record to two followers, which by design sit in the other two zones. → 2 KB cross-AZ, unavoidable if you want RF=3 across zones.
3. Consume. By default consumers fetch from the leader only. Two groups × 1 KB, two-thirds of it crossing a boundary. → ~1.33 KB cross-AZ.

That is roughly 4 KB of billed inter-zone transfer for 1 KB of data, and it scales linearly with fan-out: add a third consumer group and you add another ~0.67 KB. Most cloud providers bill inter-AZ transfer in both directions, so check your own provider's meter before modelling; the shape of the answer does not change.

The useful observation is that the 2 KB of replication is structural — it is what multi-AZ durability is — while the produce and consume crossings are largely an artifact of defaults.

Fix one: make the cluster rack-aware

Set broker.rack on every broker to its availability zone. Two things follow.

First, the partition assigner spreads replicas across racks, so a zone failure takes at most one replica of each partition rather than, with bad luck, all of them. If you are running multi-AZ without broker.rack, this is a correctness-of-your-availability-story problem before it is a cost problem — and worth checking today.

Second, broker.rack is the prerequisite for everything else below. Note that setting it on an existing cluster does not rearrange existing partitions; you need a reassignment (kafka-reassign-partitions.sh, or Cruise Control with a rack-aware goal) to converge the current topics, throttled so the rebalance does not become its own incident.

Fix two: fetch from follower (KIP-392)

Since Kafka 2.4, consumers can read from an in-sync follower in their own zone instead of the leader. It takes two settings:

• Broker: replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
• Consumer: client.rack=<the consumer's AZ>

The broker then answers each consumer's fetch with a preferred replica in the matching rack, and the consumer follows that hint. Done correctly, the consume-side crossing goes to roughly zero, which on a high-fan-out cluster is the single biggest lever available. Managed Kafka services generally support this too — on MSK the selector is a cluster configuration property; on Confluent Cloud the equivalent behaviour is handled by the platform rather than by you.

The trade-offs are real and worth stating:

• Latency floor. A follower serves only up to its high watermark, so a consumer reading locally sees data a replication round-trip later than a leader-reading consumer. For most pipelines this is single-digit milliseconds; for a latency-critical path, measure it rather than assume.
• client.rack must be correct. A wrong or missing value silently returns you to leader fetching. Inject it from the platform — instance metadata, the Kubernetes topology.kubernetes.io/zone label via the downward API — never a hand-edited config.
• Lagging followers fall back. If the local follower drops out of the ISR, fetches revert to the leader. Correct, but it means cost savings dip exactly when replication is unhealthy; do not treat the saving as a fixed number.
• Consumer version. Older clients ignore the preferred-replica hint entirely. Audit client library versions across the fleet before promising a number to finance.

Fix three: stop producers crossing zones when ordering allows

The produce-side crossing is harder, because the leader's location is a property of the partition, not the client. You have two honest options.

If your topic does not need key-based ordering, a rack-aware partitioner that prefers partitions whose leader sits in the producer's zone removes most of the crossing. This is not in the Apache client by default; it is a small Partitioner implementation reading leader metadata. The cost is skew — partition load now follows the distribution of your producer fleet, which is rarely uniform — and skew interacts badly with consumer lag and with retention sizing.

If your topic does need key ordering, leave it alone. Key-to-partition mapping is a correctness property. Trading it for network savings is the kind of clever change that reads well in a cost review and badly in a post-mortem.

The better produce-side lever is usually compression: compression.type=zstd or lz4 at the producer, with compression.type=producer on the broker so batches are stored and replicated compressed. Compression applies to all three crossings — produce, replication, and any remaining leader fetches — which makes it the only change that touches the structural 2 KB. On JSON-ish payloads a 3–5× ratio is common; on already-compact Avro or Protobuf, less. Also raise linger.ms from its default of 0 to a few milliseconds: larger batches compress better and carry proportionally less per-request overhead, at the price of a small, bounded latency increase.

What not to do

Drop to RF=2. It removes a third of replication traffic and removes most of your safety margin: with min.insync.replicas=2, one broker down blocks acknowledged writes on every partition it led. The math looks appealing on a spreadsheet and then you meet a zone outage.

Collapse to a single AZ. Single-AZ Kafka is a legitimate, deliberate architecture for some workloads — a dev cluster, or a pipeline whose data can be replayed from an upstream source of truth. It is not a cost optimisation you make quietly for a production cluster carrying core flows, because it changes your blast radius from "lose a broker" to "lose the cluster." Decide it explicitly, write it down, and make the recovery plan real.

Turn off acks=all. Different discussion, same answer: this is a durability decision, not a networking one.

The horizon: diskless and object-store-backed topics

The reason cross-AZ traffic is structural is that Kafka replicates between brokers to achieve durability. A design that writes directly to regional object storage — which is already replicated across zones by the provider — does not need broker-to-broker replication for that data. That is the premise behind tiered storage's leaderless-read direction and behind the diskless-topics work in the Apache community (KIP-1150 and related proposals), as well as several vendor implementations already in market.

Taken at face value the savings are large. Taken at the level of detail an operator needs, the trade is latency and a different failure model: object-store writes are measured in tens to hundreds of milliseconds, not single-digit, so these designs suit high-volume, latency-tolerant streams — log and telemetry ingestion, analytics feeds, replay archives — rather than the request-path topics behind a synchronous user action. Some of this is shipping, some is still proposal-stage; the honest posture today is to know which of your topics would qualify if and when it lands in your platform, and to not restructure a working cluster in anticipation.

A short audit

An afternoon's work, and usually the highest-return hour in a Kafka cost review:

1. Pull the inter-AZ data-transfer line from last month's bill and put it next to the compute line. Establish which one is bigger before optimising anything.
2. Confirm broker.rack is set on every broker and that current partition assignments are actually rack-spread — not just that the property exists.
3. Check whether RackAwareReplicaSelector is enabled and how many consumers set client.rack correctly. Fan-out-heavy clusters see the largest win here.
4. List topics by bytes-in and check compression.type on each producer, not just on the broker.
5. Identify your top three topics by volume and ask, per topic, whether it is latency-critical or replay-tolerant. That one classification drives every remaining decision, including whether diskless designs will ever matter to you.

None of this is exotic. It is configuration, a reassignment, and a willingness to read the networking section of the bill — and it reliably beats the alternative approach of buying smaller brokers and hoping.

If you want a second pair of eyes on where your cluster's bytes are going, our Kafka health check covers replica placement, client configuration, and capacity posture, and the findings come back ranked by production risk rather than by savings.