0%
Redis Deep Dive: Encodings, Memory, and Failover
The Server Model
Data Structures Inside
Memory Management
Persistence
Correct Use
Replication, the Handshake and the Backlog
A replica is an ordinary client that asks for a special thing, and the conversation it has with its primary is short enough to read in full, which is the fastest way to understand every replication failure you will ever see.
A replica is a client first
The replica opens a normal connection to the primary and speaks the normal protocol on it. There is no separate replication port, no separate authentication path, and no separate protocol dialect: the whole mechanism is built from ordinary commands, and the stream of writes that arrives afterwards is a stream of ordinary commands too.
That design decision explains a great deal of what follows. It means a replica is subject to the same client limits, the same timeouts and the same output buffer machinery as any other connection, which is why a tunable called client-output-buffer-limit can break replication. It also means the whole handshake can be performed by hand with redis-cli if you want to watch it, which is occasionally the fastest way to find out why a real replica is being refused.
The five steps
PING, to confirm the connection works and that the primary can reply. Then REPLCONF listening-port <port>, telling the primary where this replica can be reached so it can report it in INFO replication. Then REPLCONF capa eof capa psync2, declaring which protocol capabilities the replica supports. Then authentication if required. Then PSYNC <replid> <offset>.
Each step can fail and each failure looks different in the log. A PING that gets no reply is a network or firewall problem. An authentication failure after a successful PING is a masterauth problem and is extremely common after a configuration change, because the setting lives on the replica while the password lives on the primary. A refused PSYNC is a protocol or state problem, which the rest of this section is about.
What the capabilities declare
capa eof says the replica can accept a diskless transfer, where the snapshot arrives as a stream terminated by a marker rather than as a length-prefixed payload the primary had to size in advance. capa psync2 says it understands the two-identifier scheme that lets a partial resynchronisation survive a failover.
Both have been supported since Redis 4.0, so in practice every modern replica declares both and the negotiation is uninteresting. It becomes interesting during a version migration, because a primary that would like to send a diskless transfer cannot if the replica did not declare the capability, and it falls back to the disk-backed path silently. If you have configured diskless replication and the behaviour suggests it is not being used, the replica's version is the first thing to check.
PSYNC, and the question it asks
PSYNC <replid> <offset> means "I believe I was following the stream identified by replid and I had consumed up to offset; may I continue from there". A replica with no history sends PSYNC ? -1, which means "I have nothing, send me everything".
The primary now makes one decision, and it is the pivotal decision in all of replication: it either agrees to continue, or it refuses and starts the dataset from scratch. Everything expensive about replication, the fork, the transfer of the whole dataset, the memory spent buffering writes during that transfer, follows from the refusal. So the whole art of operating replication is arranging for the answer to be yes as often as possible.
The two possible answers
+CONTINUE <replid> means the primary has the requested position in its backlog and will stream from there. Nothing is transferred but the missing bytes, and the replica resumes within milliseconds. This is a partial resynchronisation and it is what you want.
+FULLRESYNC <replid> <offset> means the primary cannot or will not continue, and is about to send an entire snapshot of its dataset followed by the stream from the stated offset. On a 40GB instance that is a fork, a multi-minute transfer and a buffer that grows for its duration. This is a full resynchronisation and it is a load event on a production primary, which is why an unexplained one is worth investigating rather than shrugging at.
Why a full resynchronisation is chosen
Four reasons. The replica sent ? -1, having never synchronised or having been restarted without a usable state. The replication identifier does not match, so the replica was following a different history. The requested offset is no longer in the backlog, because too much was written during the disconnection. Or the primary has no backlog at all, because it has had no replicas for longer than repl-backlog-ttl.
The second and fourth are the ones people find surprising. A replica restarted against a primary that has been standalone for an hour gets a full resynchronisation even if it was disconnected for one second, because the primary released its backlog when its last replica left. And an identifier mismatch after a failover used to be guaranteed, which is exactly the problem the two-identifier scheme in the next section solves.
What the transfer looks like
On the disk-backed path the primary forks, writes an RDB to disk, and sends the finished file, so one child can serve several replicas that attach during the same window. On the diskless path the primary forks and writes the snapshot straight to the replica sockets, which avoids the disk entirely but historically meant one child per attaching replica.
repl-diskless-sync has been the default since Redis 7.0 and repl-diskless-sync-delay exists to make the diskless path share a child: the primary waits the configured seconds before forking so that several replicas attaching in quick succession are served by one transfer. On a deployment where replicas restart together, that delay turns three forks into one, and the default of five seconds is usually right.
How the replica loads it
repl-diskless-load controls what the replica does with the arriving snapshot. The default disabled writes it to disk and loads it, which is safe and needs disk space for a second copy. swapdb keeps the current dataset in memory while loading the new one into a second database and swaps at the end, which needs roughly double the memory but keeps the old data available if the transfer fails.
on-empty-db loads directly into memory without the backup, which is fast and cheap and loses everything if the transfer fails partway. The choice is a straightforward trade between disk, memory and safety, and the thing to notice is that the default is the conservative one: a replica with a tight disk may fail its synchronisation for reasons that look nothing like a disk problem.
The offset is the shared clock
master_repl_offset is a monotonically increasing count of bytes the primary has produced for the replication stream, and each replica's offset is how much of it that replica has consumed. The two are directly comparable because they count the same bytes of the same stream.
This makes replication unusually observable. The difference between the primary's offset and a replica's is exactly the data a promotion of that replica would lose, as the previous lesson covered, and it is also the single best health metric for a replica: a difference that stays near zero means everything is fine, and one that grows means the replica cannot keep up whatever the link status claims. No further instrumentation is needed to know the state of a replication link.
Watching a synchronisation happen
On the primary, sync_full counts full resynchronisations served, sync_partial_ok counts partial ones granted, and sync_partial_err counts partial ones refused. Those three counters are the complete history of handshake outcomes and they are cumulative, so a rate rather than a value is what to watch.
rdb_bgsave_in_progress shows the fork, and on the replica master_sync_in_progress shows the transfer with master_link_status reading down throughout. The pattern to recognise is sync_full incrementing repeatedly: that is a replica failing its synchronisation and starting over, which the fourth section covers and which is the single most damaging replication failure because each attempt costs the primary a fork and a full transfer.
What to take from the handshake
Three things. Replication is built from ordinary commands, so ordinary client machinery applies to it. The primary's decision between continuing and starting over is where all the cost lives. And both the position and the outcome are directly observable, through the offsets and through the three counters.
Everything in the rest of this lesson follows from those. The next section is about the structure that makes continuing possible, the third is about what actually travels down the stream once it is established, and the fourth is about the buffers and timeouts that decide whether an established link stays established.
A replica is an ordinary client that asks for one special thing, which is why a tunable named client-output-buffer-limit can break replication and why the whole handshake can be performed by hand with redis-cli to see why a real replica is being refused. The primary's one decision, to continue from an offset or to start over, is where every expensive part of replication lives.