Over the last two years a new class of Kafka-compatible systems has argued the same thing: the replication protocol that made Kafka durable on datacenter disks is the wrong shape for cloud object storage. WarpStream made the case first, AutoMQ and Bufstream followed, Confluent shipped Freight clusters, Redpanda shipped Cloud Topics, and the argument has now reached Apache Kafka itself as KIP-1150, "Diskless Topics" — accepted into the project and being built across several follow-on KIPs, with a per-topic opt-in rather than a new cluster type.
The pitch is easy to summarize and easy to over-read: write straight to S3, skip inter-broker replication, pay a fraction of the bill. It is a real change and it is worth understanding before a vendor explains it to you. It is also a trade, and the thing being traded is latency.
What actually changes
Classic Kafka: a partition has a leader broker, producers write to the leader, the leader writes to its local log and waits for in-sync followers to replicate before acknowledging. Durability comes from three copies on three brokers' disks, usually in three availability zones. That is why your cloud bill has a cross-AZ line item that dominates everything else, and why adding capacity means moving partitions between disks.
Diskless: there is no leader. Any broker can accept a write for any diskless topic. Each broker buffers records from many partitions for a short window — tens to a few hundred milliseconds — combines them into one object, PUTs it to object storage, and only then acknowledges the producers. Durability is the object store's problem, which is a problem it solves better than you do. A separate component (in KIP-1150's design, a batch coordinator) records where each partition's records landed inside those shared objects and assigns offsets, so consumers can still fetch an ordered partition. A compaction process later merges the interleaved objects into per-partition layouts so that reads do not degrade into scattered range requests.
Three consequences fall out of that design:
- Replication traffic disappears. Producers write once, to a regional endpoint. The three-copy fan-out across zones happens inside the object store's price, not on your network bill.
- Brokers become stateless. No local log means no partition reassignment, no data movement to scale out, no hours-long restore after losing an instance. Scaling becomes an autoscaling-group decision rather than a rebalance project.
- Latency goes from milliseconds to hundreds of milliseconds. The buffer window plus an object-store PUT is the floor. You cannot batch-and-PUT your way to a 10 ms p99.
Who the trade suits
We have run this conversation on enough engagements to have a short version.
It suits high-volume, latency-tolerant topics. Clickstream, telemetry, application logs, IoT, CDC feeds into analytics, anything whose consumer is a warehouse, a lakehouse table, or a batch job. These topics are usually the bulk of the bytes and the least sensitive to a 300 ms handoff. If a topic's data ends up in a dashboard someone looks at hourly, you are paying three-AZ replication for nothing.
It does not suit request-path topics. Synchronous flows where a user is waiting, order placement, fraud scoring, anything feeding a Kafka Streams topology whose end-to-end SLO is measured in tens of milliseconds. Diskless adds its floor at every hop, so a three-stage topology pays it three times.
It changes the capacity conversation more than the cost conversation. The bill is the headline, but the operational effect we would actually buy is statelessness: no partition rebalancing at 2 a.m., no waiting for a replacement broker to catch up, no sizing disks eighteen months ahead. For teams whose Kafka pain is operational rather than financial, that is the stronger argument.
What you are signing up for
Honest accounting requires the other column.
Object-store request costs are not zero. The saving is in cross-AZ transfer, not in API calls. A design that PUTs frequently to keep latency down issues a lot of PUTs, and GETs on the read side scale with consumer fan-out and cache misses. Model it with your own numbers: bytes per second, producer count, consumer groups, and how many of those consumers read historical data. Vendors' calculators assume their own batching behaviour.
A new metadata component is on the critical path. Whatever assigns offsets and records object layout — batch coordinator, metadata service, however a given implementation names it — is now a thing that can be slow, can be a bottleneck under high partition counts, and has to be operated or trusted to a vendor. Ask how it is replicated and what its failure mode looks like. "Durability is S3's problem" is true about bytes and not about metadata.
Compaction is real work. Interleaved objects have to be reorganized into per-partition files or historical reads get expensive. That background process consumes CPU and request budget, and if it falls behind, read latency and cost drift upward quietly. It is the same operational lesson as tiered storage: the new tier is not free of operations, it has different ones.
Feature coverage lags. Early diskless implementations vary in support for compacted topics, transactions and exactly-once semantics, and cross-cluster replication. If a topic is an EOS input or a compacted changelog, check rather than assume. For Apache Kafka's own KIP-1150 work, note that it is a topic-level setting inside a normal KRaft cluster: same brokers, same protocol, same clients, some topics diskless and some not. That mixed model is the most useful part of the design, because it lets you move the cheap traffic and leave the latency-sensitive traffic alone.
How we would approach it
Nothing here justifies a migration project on its own. What it justifies is a measurement.
- Rank your topics by bytes per day. In most estates we review, the top three to five topics are 80% of the volume. Those are the only candidates that matter.
- For each candidate, write down the actual end-to-end latency requirement — not the number in the design doc, the number a consumer would notice. Most analytics-bound topics have no one to notice.
- Measure current cross-AZ spend attributable to those topics. Replication traffic is roughly (replication factor − 1) × produced bytes crossing zones, plus consumer fetches that are not rack-aware. If you have not enabled follower fetching yet, do that first: it is a config change, it is reversible, and it may capture enough of the saving that the rest of this can wait.
- Price the diskless version with request costs included, for at least one open implementation and one managed one. The gap between the two is usually the real decision.
- Run one topic. Not the busiest, not the most critical — a high-volume, low-stakes one, in parallel via a mirrored write if you can afford it, for long enough to see what a full retention cycle and a consumer backfill cost.
The technology is sound and the direction is clear: object storage is where high-volume logs belong, and Kafka the protocol is outliving Kafka the storage engine. But the same estates that would benefit most usually have simpler savings sitting untouched — rack-aware fetching, retention that reflects real replay needs, partition counts that were guessed in 2021. Diskless is a good tool. It is rarely the first one to reach for.
If you are weighing an object-storage-backed Kafka against your current cluster, we do this as a short engagement: measured traffic, a per-topic latency inventory, and a cost model with the request charges in it. Tell us what your top topics carry and we will tell you which of them, if any, belong on object storage.