0%
Kafka Deep Dive: Logs, Brokers, and Exactly-Once
The Log Abstraction
Storage Internals
Cluster Coordination
Producers
Consumers
Transactions
Multi-Cluster and Geo-Replication
A second Kafka cluster is not a feature of Kafka. It is a separate cluster with its own brokers, its own offsets and its own controller, and connecting the two is an application problem that MirrorMaker 2 solves by being a set of Connect connectors rather than anything built into the broker.
What MirrorMaker 2 actually is
Three connectors on a Connect cluster, which is the fact that explains most of its behaviour.
MirrorSourceConnector consumes from the source cluster and produces to the target, one task per partition group. MirrorCheckpointConnector reads consumer group offsets on the source and writes translated checkpoints to the target. MirrorHeartbeatConnector produces a record into a heartbeats topic on a schedule, which is how liveness and end-to-end lag are measured.
Because they are Connect connectors, everything from the Connect lesson applies: the tasks are rebalanced by the Connect group protocol, their progress is stored in Connect's offset.storage.topic, a failing task is retried according to Connect's error handling, and tasks.max is the parallelism control. A MirrorMaker 2 problem is usually a Connect problem, and people debug it as a Kafka problem.
It can be run three ways: as connectors on an existing Connect cluster, as a dedicated MirrorMaker 2 cluster with connect-mirror-maker.sh, or embedded in an application. The dedicated mode is the common choice and runs its own simplified Connect runtime, which means it does not expose the full Connect REST API, and that difference surprises people trying to inspect it.
Where to run it, which is not arbitrary
At the target, consuming across the link rather than producing across it.
The asymmetry is real. A producer writing across a high-latency link holds records in its accumulator and is sensitive to the link's latency through delivery.timeout.ms, and a failed produce means the record must be retried from the producer's buffer, which is bounded by buffer.memory. A consumer reading across a link is simply slow: the fetch takes longer, the data arrives, and nothing is at risk because the source retains it.
So remote-consume beats remote-produce, and the rule is that MirrorMaker 2 runs in the same region as the cluster it writes to. For bidirectional replication that means two MirrorMaker 2 deployments, one in each region, rather than one in the middle.
The second reason is failure semantics. If the link drops, a target-side MirrorMaker 2 stops consuming and resumes where it left off, with the source's retention as the bound on how long an outage is survivable. A source-side deployment with a full buffer starts dropping records, which converts a network outage into data loss.
The alternative of writing to both clusters
Proposed often enough to be worth dismissing precisely.
Instead of replicating, have each producer write every record to both clusters. It sounds simpler, requires no Connect cluster, and gives both clusters the data at the same moment. It is wrong for three reasons.
A dual write is not atomic. The first produce succeeds, the second fails, and the application now holds a record that exists on one cluster and not the other with no mechanism to reconcile. Retrying the second write means the first cluster's copy is older, and giving up means permanent divergence.
The offsets diverge anyway. Two independent clusters assign offsets independently, so the two copies of a record sit at different offsets, and every problem in the offset translation section returns without the sampled anchors that made it tractable.
And it puts the cross-region link in the produce path, which is the latency cost a stretch cluster pays and the thing replication exists to avoid. A regional link degradation now degrades the application rather than the replication lag.
Replication is worse in one respect, a non-zero lag, and better in every other, which is the trade MirrorMaker 2 exists to make.
The configuration shape
Declarative, naming clusters and the flows between them.
The topics value is a regular expression, not a list, which is the single most consequential detail in the file. A pattern of .* replicates every topic including the internal ones, and the resulting feedback behaviour is the classic first MirrorMaker 2 incident. The topics.exclude default already excludes internal topics and a hand-written override loses that protection.
refresh.topics.interval.seconds, default 600, is how often new topics matching the pattern are discovered, which means a newly created topic is not replicated for up to ten minutes. Teams that create topics programmatically and expect immediate replication find this out during a migration.
Sizing the replication flow
Three numbers, and the first one is the only one most people set.
tasks.max bounds the number of tasks, and the source connector distributes partitions across them, so a flow replicating 2,000 partitions with tasks.max=16 gives each task 125 partitions to consume and produce. Each task is a consumer and a producer in one thread, which means a task's throughput is bounded by the round trip of its produce to the target, and that round trip is local because MirrorMaker 2 runs at the target.
The second number is the producer configuration applied to the target-side produce, set with the <target>.producer.override.* prefix. The defaults are the ordinary producer defaults, which are wrong for a replication flow: batch.size at 16,384 and linger.ms at 0 produce the same small-batch inefficiency described in the producer lesson, except here the input is already batched and the whole workload is throughput rather than latency. Raising linger.ms to 50 or 100 and batch.size to 262,144 is standard, and compression.type matters only if the source's records were not already compressed, because a recompression is pure cost.
The third is the consumer configuration on the source side, <source>.consumer.override.*, where fetch.min.bytes and max.partition.fetch.bytes govern how much arrives per cross-region round trip. On a high-latency link this is the setting that matters most: a 100 ms round trip fetching 1 MB moves 10 MB per second per task, and fetching 10 MB moves 100 MB per second per task, from the same number of round trips.
Measuring whether it is keeping up
Two measurements, and the obvious one is the weaker.
The obvious one is MirrorMaker 2's own Connect metrics, which report records and bytes replicated per second per task and the source connector's own lag. They are useful for capacity and they do not answer the question that matters, because they are a view from inside the replication rather than from end to end.
The one that answers it is the heartbeat. MirrorHeartbeatConnector produces a record carrying a source-set timestamp into a heartbeats topic, which is then replicated like any other topic, so a consumer on the target reading <source>.heartbeats can subtract the record's timestamp from now and obtain the replication lag of the entire chain: the source's produce, the connector's consume, the link, the target's produce, and Connect's own scheduling delays.
That number is the one a disaster recovery plan is built on, which is the argument for enabling heartbeats on every flow even when nothing else consumes them. The cost is one record per interval per flow, which is nothing.
The third measurement, worth having and often skipped, is a comparison of log-end-offset between the source and target for the same partition, available from either side's admin client. It is not a lag in time, and it catches the specific failure where a flow is running, its metrics look healthy, and one partition has stopped moving because its task is wedged.
What it replicates and what it does not
Four things it carries, three it does not.
It carries records, byte for byte including headers and timestamps, so a record's CreateTime survives. It carries topic configuration, synced on sync.topic.configs.enabled, which is why a retention change on the source propagates. It carries ACLs under sync.topic.acls.enabled, with the caveat that write ACLs are deliberately not propagated to a target where the topic is read-only. And it carries consumer group offsets as checkpoints, which is the subject of a later section.
It does not carry partition assignment of keys in a useful sense: records land in the same partition number, which preserves per-key ordering only because the partition index is copied rather than recomputed. Change the target's partition count and that guarantee is gone.
It does not carry transactions. A transactional producer's records are replicated as ordinary records, markers are not meaningfully reproduced on the target, and a read_committed consumer on the target does not see source transaction boundaries. Exactly-once does not cross a MirrorMaker 2 link, and any design that assumes it does is wrong in a way that is hard to see.
And it does not carry schema registry state, which is a separate system with its own replication problem and its own section later in this lesson.
MirrorMaker 2's topics setting is a regular expression rather than a list, and the most common first incident is a pattern of .* replicating internal topics including __consumer_offsets, which then interacts with checkpoint writes in ways that are difficult to unpick. The topics.exclude default already excludes internal topics, so a hand-written override silently removes that protection. Write the pattern as an explicit alternation of the topics intended, and treat a wildcard as a decision requiring review.