MirrorMaker 2 in Practice: What Cross-Cluster Replication Actually Guarantees

Cross-cluster replication is one of the easiest things to switch on in Kafka and one of the hardest to rely on. A MirrorMaker 2 connector takes an afternoon. The question that matters — if the primary cluster goes away right now, where do our consumers resume, and what do they see twice? — usually goes unanswered until the day it is asked in anger.

This is a field note on what MirrorMaker 2 (MM2) actually gives you, where the guarantees stop, and the rehearsal that converts a replication setup into a failover plan.

What MM2 is

MM2 is a set of Kafka Connect connectors, not a separate service:

• MirrorSourceConnector copies records from remote topics into local topics, and mirrors topic configuration and ACLs.
• MirrorCheckpointConnector reads __consumer_offsets on the source, translates group offsets into target-cluster offsets, and writes them to a <source>.checkpoints.internal topic — and, if sync.group.offsets.enabled is on, into the target cluster's __consumer_offsets for groups that are not currently active there.
• MirrorHeartbeatConnector produces a steady heartbeats topic, which is how you measure whether the link is alive independently of whether your business topics have traffic.

You can run these inside a dedicated Connect cluster (our default for anything production-critical, because it gives you the normal Connect controls: separate scaling, REST API, per-connector metrics) or via the bundled connect-mirror-maker.sh driver. The driver is convenient for a migration; a long-lived replication link deserves a real Connect cluster.

Records are copied. Offsets are not the same number.

This is the single most important mechanical fact. MM2 produces into the target cluster as an ordinary producer. Target partitions therefore have their own offsets, and they will not match the source. Retention on the source, a restarted connector, a compacted topic, a partition that existed before replication started — any of these shift the mapping.

Ordering within a partition is preserved, because MM2 maps source partition n to target partition n and writes in order. Global ordering across partitions is not preserved, and never was in Kafka.

Delivery is at-least-once. MM2 can re-deliver records after a connector restart or a rebalance of the Connect worker. Exactly-once support exists (exactly.once.source.support on the Connect cluster plus a transactional MM2 source), but it costs throughput and adds a transactional coordinator to the failure path. Assume duplicates and make the consuming side idempotent unless you have deliberately paid for something else.

Offset translation is approximate, and approximately is enough only if you know which way it errs

Checkpoints let a consumer group that was reading orders on cluster A resume near the right place on us-east.orders on cluster B. Two limits to internalise:

1. Translation is checkpoint-granular. Checkpoints are emitted on an interval (emit.checkpoints.interval.seconds, 60s by default). The translated offset is the last one MM2 could map, so it lags the true committed offset. On failover, groups resume slightly behind where they were. That is the correct direction to err — duplicates, not gaps — but it means a failover reprocesses some records, and that window is roughly your checkpoint interval plus replication lag.
2. Offsets are only synced for inactive groups. sync.group.offsets.enabled deliberately refuses to overwrite offsets for a group that has live members on the target cluster. This is a safety feature and it is also why "we enabled offset sync" does not mean "our consumers can start on the DR cluster." If a warm consumer group is already running there, it owns its own offsets.

If your clients are new enough, RemoteClusterUtils / the MirrorClient API lets a consumer ask for translated offsets explicitly at startup, which is cleaner than relying on the sync to have landed.

Topic naming: pick a policy once, and know what it costs

The default DefaultReplicationPolicy prefixes replicated topics with the source alias: orders becomes us-east.orders. This is what makes active/active safe — the prefix identifies provenance, so MM2 can refuse to replicate a topic back to where it came from, and cycles cannot form.

IdentityReplicationPolicy keeps the original name, which is what most migration projects want: consumers move clusters without touching their subscriptions. The trade-off is real and non-negotiable — identity naming gives up automatic loop prevention, so it is safe for one-directional replication (a migration, a read-only DR copy) and not safe for a bidirectional topology. Choose the policy before you create topics; changing it later means renaming topics in production, which in practice means a new pipeline and a cutover.

Active/active, active/passive, and being honest about which you have

Active/passive is the common and defensible setup: one cluster takes writes, the other holds a replica that nobody reads. It is simple to reason about, and its weakness is that the passive side is untested by definition. Every capability you have not exercised there — ACLs, quotas, Schema Registry contents, client DNS, connector state — is an assumption.

Active/active means producers write in both regions and each side replicates the other's topics in. It works, with the default prefixing policy, and it pushes the hard problem into the application: a consumer that must see the global stream subscribes to a pattern covering both local and remote topics, and must tolerate interleaving that has no single global order. If your logic requires a total order across regions, active/active will not give you one, and no replication tool will.

What almost nobody has, and many teams believe they have, is synchronous cross-region replication. MM2 is asynchronous. A regional failure loses whatever was in flight. The relevant number is your replication lag at the moment of failure, and the only way to know it is to measure it continuously.

Metadata is the part that bites

Records replicate. Several things around them do not, or do so with caveats:

• Schema Registry is a separate system with its own storage topic. Replicating _schemas naively into a target registry that is also writing to it produces a broken registry. Plan registry replication explicitly — a dedicated one-way copy, or a shared registry with a documented write path.
• Consumer group state (members, assignments, generation) does not replicate. Only translated offsets do.
• Transactional state does not replicate. A stream of records that was written transactionally on the source arrives on the target as plain records; read_committed semantics do not carry across the link.
• Kafka Streams state stores do not usefully replicate. Changelog topics can be copied, but a Streams application failing over will generally restore from the target's changelog and that restore time is your real RTO — measure it.
• ACLs and topic configs replicate if you enable sync.topic.acls.enabled and sync.topic.configs.enabled. Check that you want that; a target cluster with different principals or different retention economics may not.

What to monitor

Four things, in priority order:

1. End-to-end lag, from the heartbeats topic: produce time on the source versus arrival time on the target. This works even when business topics are idle, which is exactly when you would otherwise be blind.
2. replication-latency-ms and record-age-ms per topic-partition from the MM2 source connector metrics. Per-partition is the granularity that matters; an average hides the one partition whose leader moved.
3. Connect task state. A failed task is silent unless you alert on it. FAILED tasks in a MM2 Connect cluster are the most common cause of "our DR copy was three days stale."
4. Checkpoint freshness — the age of the newest checkpoint per group. That number is your failover reprocessing window, and if it grows, your RPO grew without anyone deciding to let it.

The rehearsal

Replication you have not failed over to is a backup you have not restored. A rehearsal we would run, in a staging estate that mirrors production topology:

1. Note the current replication lag and the newest checkpoint age for every group in scope.
2. Stop producers to the source cluster. Let the link drain, and record how long draining takes — in a real incident you will not get this pause, so this is your best case.
3. Stop the consumer groups on the source. Read the translated offsets for each group from the checkpoints topic or via MirrorClient.
4. Start the consumer groups against the target cluster, with the topic names the target actually uses (this is where an unexamined replication policy shows up).
5. Measure: how many records were reprocessed per group, how long until each group caught up, and whether any downstream system double-counted. The last one is the finding that matters — it tells you which consumers were not idempotent after all.
6. Fail back, and measure again. Fail-back is usually messier than failover, because the primary's topics now diverge, and teams that skipped this step end up rebuilding a cluster by hand.

Write the numbers down. "Our RPO is under a minute and our RTO is eleven minutes for the payments group, dominated by Streams state restore" is a sentence you can plan with. "We have MirrorMaker running" is not.

When MM2 is the wrong tool

For a one-time migration between clusters where you control the cutover window, MM2 plus identity naming is a good fit. For a permanent multi-region topology, look honestly at the alternatives first: a stretch cluster with rack-aware replicas across nearby availability zones gives you synchronous durability that MM2 cannot, at the cost of inter-zone latency and a tighter failure domain. Managed cluster-linking products replace MM2's offset translation with byte-for-byte offset preservation, which removes a whole class of problems in exchange for platform coupling. And for some estates the correct answer is that the disaster-recovery requirement is really a data requirement, better served by a durable object-store copy than by a second live cluster.

The decision is about which failure you are buying insurance against, and at what operational cost. If you are standing up replication now, or you have a link running that nobody has ever failed over to, describe the topology and we will tell you what we would test first.