Kafka's replication is a leader-follower scheme with one unusual property: followers are ordinary consumers. They issue fetch requests against the leader, receive record batches, and append them to their own log. There is no separate replication protocol, no push, and no consensus per record. Understanding why that choice was made, and what it costs, is most of understanding Kafka's availability behaviour.

Followers pull rather than being pushed to

A follower sends a Fetch request naming each partition it replicates and the offset it wants next. The leader responds with batches from that offset onward, byte-for-byte as they sit on its disk, which means a follower fetch can use the same sendfile path as a consumer fetch.

The pull model has three advantages that the push model does not. The follower controls its own rate, so a slow follower falls behind rather than overwhelming itself or forcing the leader to buffer per-follower state. The leader needs no knowledge of follower progress beyond what the fetch requests tell it, so there is no per-follower queue to manage. And the code path is the consumer path, hardened by every consumer in existence.

Its cost is latency. A record appended at time T is not replicated until a follower's next fetch returns, so replication latency is bounded below by the fetch round trip rather than by the leader's willingness to send. replica.fetch.wait.max.ms, 500 by default, caps how long the leader holds a fetch waiting for data, and replica.fetch.min.bytes at 1 means it returns as soon as anything is available, so in practice a follower is a round trip behind rather than half a second.

The three offsets every replica tracks

Precision here pays for itself repeatedly, because almost every replication question is a question about one of these.

The log end offset, or LEO, is the offset that will be assigned to the next record appended. On the leader it advances on every append. On a follower it advances as it appends what it fetched.

The high watermark, or HW, is the highest offset that every in-sync replica has. The leader computes it as the minimum LEO across the in-sync set and advances it as followers catch up. It is the boundary of committed data.

The log start offset is the lowest offset still present, which moves forward as retention deletes segments or as deleteRecords is called.

What the high watermark means to a consumer

Consumers cannot read above the high watermark. A record appended to the leader but not yet replicated to the in-sync set is physically present on the leader's disk and invisible to every consumer.

The reason is the only reason that matters: a record above the high watermark might not survive. If the leader fails before that record is replicated, a new leader elected from the in-sync set will not have it, and the record is gone. Had a consumer been allowed to read it, that consumer would have processed a record that the cluster subsequently decided never existed, which is a correctness failure no application can defend against.

So the high watermark is Kafka's commit point, and the gap between LEO and HW is the window of written-but-not-committed data. In a healthy cluster that gap is one fetch round trip's worth of records, typically a few milliseconds. Under stress it grows, and a growing gap is a direct measure of replication falling behind.

How the high watermark propagates back

The leader knows the high watermark immediately; followers have to be told, and the mechanism is worth knowing because it creates a small asymmetry.

The leader includes the current high watermark in every fetch response. So a follower learns the new high watermark on its next fetch after the one that delivered the data, which means a follower's view of the high watermark is always at least one round trip stale.

This matters during leader election. A newly elected leader starts with its own possibly stale high watermark and advances it as it learns the new in-sync set's progress, which is why there is a brief period after an election where the committed boundary is conservative. It also matters for fetch.from.follower reads, where a consumer reading from a follower sees a slightly older committed boundary than one reading from the leader.

Each replica also checkpoints its high watermark to disk every replica.high.watermark.checkpoint.interval.ms, 5,000 by default, so that a restarting broker does not have to treat its whole log as uncommitted.

Replication is not consensus, and that is deliberate

It is worth being explicit about what Kafka's replication is not, because the comparison is often made loosely.

Raft and Paxos commit a record once a majority has it. Kafka commits once the in-sync set has it, and the in-sync set is a dynamic membership maintained by the controller rather than a majority of a fixed set. With three replicas all in sync, Kafka requires all three, which is stricter than a majority. With two replicas having fallen out of the in-sync set, Kafka requires only the one remaining, which is far weaker.

That dynamism is the whole design. It buys availability, because a partition with one surviving replica keeps serving. It costs the clean guarantee a majority quorum gives, because the durability of a committed record depends on how large the in-sync set happened to be at the time. min.insync.replicas exists precisely to put a floor under that, and configuring it is how an operator chooses where on the availability-durability line the partition sits.

The settings that govern follower fetching

Six, and the first two are the ones that matter on a busy cluster.

num.replica.fetchers, 1 by default, is the number of fetcher threads per broker, each handling all partitions fetched from one leader. At 1, a broker replicating from six leaders has six fetcher threads but each handles every partition from that leader serially. On a broker with thousands of replicas this is the commonest cause of replication falling behind, and 4 to 8 is a reasonable setting.

replica.fetch.max.bytes, 1,048,576, caps bytes per partition per fetch, and replica.fetch.response.max.bytes, 10,485,760, caps the whole response. A partition whose single record batch exceeds the per-partition limit is still returned, so a large message cannot stall replication, but the limits do bound catch-up throughput.

replica.fetch.wait.max.ms at 500 and replica.fetch.min.bytes at 1 together mean a fetch returns immediately when data exists and waits up to half a second when it does not. replica.socket.receive.buffer.bytes, 65,536, is the socket buffer, and on a high bandwidth-delay-product link it is too small, which matters for cross-region replication.

Why a follower's fetch is also a consumer

One consequence of the shared code path deserves its own note, because it explains a class of capacity surprise.

A follower fetch consumes the same resources as a consumer fetch: a request handler thread, a slot in the request queue, and page cache for the bytes it reads. A broker leading 5,000 partitions with replication factor 3 is serving 10,000 follower fetches' worth of work in addition to its clients.

That is why replication traffic shows up in the same metrics as client traffic and why a broker that becomes a leader for many partitions after a failure can be overloaded by the replication alone. It is also why BytesOutPerSec on a healthy cluster is roughly client egress plus two times ingest, and knowing that decomposition makes the metric interpretable rather than puzzling.

Replication throughput against produce throughput

The two are related by a factor people forget, and the omission shows up as a cluster that works at half its planned ingest.

A broker accepting X MB/s of ingest at replication factor 3 must also serve 2X MB/s of outbound replication, because each of the two followers reads everything it wrote. Add client consumption and a broker's outbound traffic is two times ingest plus consumer egress, before any catch-up or reassignment.

So a 10 Gb/s network interface, about 1.25 GB/s, carries roughly 400 MB/s of ingest at replication factor 3 with one consumer group reading everything, and that is the ceiling before any headroom for recovery. Planning from ingest alone overstates capacity by a factor of three or four.