Most Kafka estates we review now have a second consumer of everything: the lakehouse. Analysts want the same events the microservices get, in Apache Iceberg, queryable from Spark, Trino, Snowflake, or DuckDB. The question is no longer whether to land topics as tables — it is which component performs the handoff, and what that choice commits you to operationally.
There are three common answers. None is wrong; they fail in different places.
The shape of the problem
A Kafka topic is an append-only log partitioned by key, with retention measured in hours or days. An Iceberg table is a set of immutable data files plus a metadata tree, updated by commits that swap the current snapshot. Getting from one to the other means answering four questions:
- Commit cadence. Every commit creates a snapshot and a manifest. Commit every 10 seconds and you get 8,640 snapshots a day and a metadata tree that makes planning slow. Commit every 15 minutes and your "real-time" table is 15 minutes stale. Freshness and metadata health pull against each other, and the commit interval is where you set the exchange rate.
- File sizing. One writer task per partition, committing frequently, produces files of a few megabytes. Query engines want 128–512 MB. Small files are not a cosmetic problem: they inflate planning time, manifest size, and object-store request bills. Something must compact, and you must own that something.
- Delivery semantics. Kafka gives you at-least-once cheaply. Iceberg tables that double-count are wrong in the same permanent way a Streams aggregate is. The writer needs to tie its Kafka offset commit to the Iceberg snapshot commit, or you need idempotency on a business key downstream.
- Schema evolution. The topic's schema will change. Iceberg supports column add, drop, rename, and reorder by field id — but only if the writer maps registry schema changes onto Iceberg schema updates rather than failing, or worse, silently dropping the new column.
Every option below is a different set of answers to those four.
Option 1: Kafka Connect with an Iceberg sink
The Iceberg sink connector (contributed from the Tabular work, now part of the Iceberg project) runs on your existing Connect cluster. It coordinates commits across tasks through a control topic: workers write data files, a coordinator collects them and performs a single Iceberg commit, then offsets advance. That coordination is what makes the exactly-once-ish story work — the snapshot records the offsets it contains, so a restart resumes from the committed point rather than duplicating.
What it's good at. You already run Connect, so it is one more connector in a deployment model your team knows: same REST API, same connect-offsets topic, same rolling-restart procedure. It handles multi-table fan-out from one topic and can auto-create tables.
What bites. Everything in running Connect in production applies here, plus two specifics. First, the commit interval is a connector setting and it is the freshness/metadata trade-off named above — pick it deliberately, not by leaving the default. Second, the sink writes files; it does not maintain the table. Compaction, snapshot expiration, and orphan-file cleanup are separate jobs you schedule yourself (Spark procedures, or your catalog vendor's maintenance service). Teams that skip this discover it three months later when Trino planning takes 40 seconds on a table with 200,000 tiny files.
Option 2: Flink (or Spark Structured Streaming) writing Iceberg
If you already run Flink for stream processing, its Iceberg connector is mature and the checkpoint barrier gives you a natural commit boundary: Flink commits an Iceberg snapshot per checkpoint, atomically with its own offset state. Set the checkpoint interval and you have set the commit cadence, with exactly-once semantics that come from the engine rather than a connector-specific protocol.
What it's good at. Transformation on the way in — filtering, joins, enrichment, flattening a nested event into an analyst-shaped row. Upserts into Iceberg v2 tables with equality deletes, which is how CDC streams become mirror tables rather than append-only change logs. Flink also ships table maintenance actions you can run in the same cluster.
What bites. You are adopting a distributed stream processor to move bytes. If Flink is not already in your estate, the operational bill — checkpoint storage, state backend tuning, job restarts, a second scheduler to run on call for — is larger than the problem. We have talked more than one team out of this. If Flink is already there, it is usually the right answer, because one commit protocol beats two.
Option 3: Managed materialization at the broker layer
The newer category: the streaming platform itself exposes topics as Iceberg tables. Confluent Tableflow, Redpanda's Iceberg Topics, AutoMQ's table topics, and similar features in cloud offerings all take the position that if the log already lives on object storage, a second copy for analytics is waste. You mark a topic, the platform maintains the table and its metadata, including compaction.
What it's good at. It removes a moving part you would otherwise own. No connector fleet, no maintenance cron, no separate offset story. For teams whose lakehouse need is "the same events, queryable, without a data-engineering headcount," this is a genuinely smaller system.
What bites. It is the least vendor-neutral choice on the list, and we say that as people who will happily recommend it anyway. The feature is tied to a platform, its catalog integration set is whatever the vendor supports this quarter, and transformation options range from thin to none — what lands is broadly what was on the topic. Ask three questions before committing: which catalogs it can register with (Glue, Polaris, Unity, REST), whether you can read the table with your own engines without paying per-query to the vendor, and what happens to the table if you turn the feature off or leave the platform. If the answer to the last one is "the data stays in your bucket in open Iceberg format," the lock-in is modest. If it is not, price that in.
Deciding
A short version of how we reason about it on engagements:
- Flink already in production, or CDC upserts needed. Use Flink. Checkpoint-aligned commits are the cleanest semantics available, and equality deletes are hard to get elsewhere.
- Connect already in production, append-only events, no transformation needed. Use the Iceberg sink connector, and schedule compaction and snapshot expiry in the same change that deploys the connector. Not later.
- No stream-processing platform, and your vendor offers materialization on open Iceberg files in your own storage. Use it. Owning less is a real benefit and the escape hatch is the file format.
- Fewer than a handful of topics and hourly freshness is fine. Consider not streaming at all: a scheduled batch job reading the topic with a bounded consumer is boring, cheap, and nobody is on call for it.
The parts everyone forgets
Whichever path you take, three things are yours regardless of the component:
Partitioning strategy on the table side. Iceberg partitioning is not Kafka partitioning. Hidden partitioning on an event-time column with day transform is the usual starting point; partitioning by ingest time produces tables that are fast to write and slow to query. If event time can arrive late, understand which partitions get rewritten and how often.
Table maintenance as a monitored job. Compaction, expire_snapshots, and orphan-file removal should have the same alerting you would give any other production job. A silent maintenance failure looks fine for weeks and then presents as "the lakehouse got slow."
Schema change rehearsal. Add a nullable field to the topic's schema in staging and watch what the table does. Under transitive compatibility rules the registry will accept the change; whether your writer propagates it as an Iceberg column add, ignores it, or stops is a property of the writer, not the registry. Find out on a Tuesday afternoon rather than during a release.
The handoff between a stream and a table is one of the few places where a Kafka estate touches a completely different operational culture. Pick the component that fits the on-call rotation you actually have, write down the commit cadence and who compacts, and the lakehouse becomes what it should be: another consumer group, just a slow one.
If you are choosing between these — or already have an Iceberg sink quietly producing small files — a health check or an architecture review is a short engagement with a written recommendation at the end. Tell us about the pipeline.