Tiered storage arrived in Apache Kafka as KIP-405 — early access in 3.6, production-ready in 3.9, and carried into 4.0 — after several years of vendor-specific implementations on MSK and Confluent. The mechanism is simple to state: once a log segment is closed, the broker uploads it to remote storage (S3 or an equivalent) and eventually deletes the local copy, while the offsets stay addressable as if nothing moved. Consumers reading recent data are served from local disk; consumers reading old data trigger a remote fetch.
That one change touches broker sizing, rebalance duration, retention policy, and read latency at the same time, which is why it is worth deciding deliberately rather than turning on because it exists.
What it actually changes
Before tiered storage, retention and disk were the same decision. Thirty days of a 200 MB/s topic at replication factor 3 is roughly 1.5 PB of broker disk, so most teams set retention to whatever their disks could hold — commonly seven days — and told anyone who needed history to go read the warehouse instead.
With tiering, the two decisions separate:
remote.storage.enable=trueon the topic turns on offload.local.retention.ms(orlocal.retention.bytes) governs how much of the log stays on the broker.retention.msnow governs the total log, local plus remote.
So a topic can keep four hours locally and ninety days in total. Broker disks get sized for the local window plus headroom, not for the retention promise.
The second-order effect is the one operators feel first: replica movement gets cheap. Adding a broker, replacing a failed one, or rebalancing partitions only copies the local window. A partition that used to drag 2 TB across the network during reassignment now drags 40 GB. Cluster expansion goes from an all-weekend exercise with a throttle spreadsheet to something you do on a Tuesday afternoon. For teams whose real problem is "we cannot safely grow this cluster," that benefit alone often outweighs the storage bill.
What it costs
Cold reads are slower and noisier. A remote fetch pulls from object storage, so a consumer starting at the beginning of a tiered topic sees first-byte latency in the hundreds of milliseconds rather than single-digit ones, and throughput bounded by remote-read concurrency (remote.fetch.max.wait.ms, the remote log reader thread pool) rather than page cache. A backfill that used to take twenty minutes may take an hour. That is usually fine — but only if you know which consumers do cold reads.
Request cost, not just storage cost. Object storage is roughly an order of magnitude cheaper per byte than provisioned block storage, and you stop paying for it three times over via replication. But GET requests and — on cloud providers that charge it — cross-AZ or egress traffic are billed per operation. A pipeline that repeatedly replays a long history from remote storage can spend more on requests than it saved on bytes. Model both terms with your actual replay pattern before quoting a number to finance.
A new component in the failure path. The remote log manager, its metadata topic (__remote_log_metadata by default), and the plugin implementation all become things that can be unhealthy. Offload can fall behind; if it does, local segments cannot be deleted and disks fill anyway. Monitor the offload lag and the copy/fetch error rates as first-class signals — RemoteCopyLagBytes, RemoteCopyLagSegments, RemoteFetchErrorsPerSec and their siblings — not just disk usage.
Restrictions that surprise people. Compacted topics are not supported, so your key-latest state topics and Kafka Streams changelogs stay on local disk. Tiering also cannot be cleanly reversed on a live topic without a plan for the data already remote. And once history is retrievable, it is discoverable: ninety days of retained events is ninety days of data subject to whatever deletion obligations you carry. Check that against your retention policy on purpose rather than discovering the conflict during an audit.
Where it earns its place
Four situations where we have seen tiering clearly pay:
- Replay is part of the design. Event-sourced systems, ML feature backfills, or a new consumer that must rebuild state from the topic rather than a database. Tiering turns "we keep seven days because disks" into a retention decision the architecture can actually rely on.
- The cluster is hard to grow. Large per-broker footprints make every reassignment a risk. Shrinking the local window shrinks the risk.
- Retention was set by disk, not requirement. If someone can state the business reason for thirty or ninety days and the only obstacle was storage, this is the tool.
- Incident forensics matter. Being able to re-read the exact bytes from three weeks ago changes the quality of a post-mortem.
And the cases where it does not:
- Retention is already short by requirement — a topic feeding a real-time join with a two-hour window gains nothing.
- The topic is compacted; unsupported.
- Cold reads are the normal access pattern, not the exception. If most consumption is historical, you are building a data lake with a Kafka API in front of it, and a lake with a table format is likely the cheaper and better-suited answer.
A rollout that does not surprise anyone
- Pick one topic that is high-volume, non-compacted, and not latency-critical. Ideally one whose disk usage you already resent.
- Measure first. Current retention, bytes/day per partition, per-broker disk, and — critically — which consumers ever read older than a few hours. Consumer start offsets over the last month answer that.
- Set local retention generously to begin with, well above the oldest offset your live consumers touch, so no normal consumption path becomes a remote fetch on day one.
- Enable offload and watch the copy lag until it is steady and segments begin expiring locally. Confirm disk actually falls; if it does not, offload is behind or
local.retention.*is not what you think. - Extend total retention only then. Test a deliberate cold read — a consumer group reset to the earliest offset on a test group — and record the throughput you get, so the next backfill is planned with a real number.
- Tighten local retention gradually, checking the remote fetch rate after each step. The moment it climbs during normal operation, you have gone one step too far.
On MSK and Confluent Cloud the same trade-offs apply behind managed knobs: the vendor operates the offload path, you still choose the local window and still pay for cold reads in latency and requests. The judgment does not change with the operator; only who gets paged when offload stalls.
The short version
Tiered storage decouples retention from broker disk, and its quietest benefit — fast replica movement — is often the one that matters most operationally. It is not free: cold reads are slower, request costs are real, and there is one more subsystem to monitor. Decide it per topic, from measured consumer behaviour, and roll it out with local retention wide before you narrow it.
If you are weighing tiered storage against a shorter retention window or an external archive, tell us what the topic carries and we will work through the sizing and cost model with you.