0%
Redis Deep Dive: Encodings, Memory, and Failover
The Server Model
Data Structures Inside
Memory Management
Replication and Cluster
Correct Use
Recovery, and the Edges of What Redis Promises
Persistence settings are a means; the ends are two numbers, how long a recovery takes and how much data it loses, and almost nobody running Redis knows either one for their own deployment.
The two numbers
Recovery time is how long until the instance is serving again. Recovery point is how far back in time the data has been rolled, which is the loss window the previous lesson was about. They are independent: a configuration can recover quickly and lose a lot, or slowly and lose nothing.
Stating both turns persistence from a matter of taste into a specification. A team that says "we can be down for five minutes and we cannot lose more than a second" has described a configuration almost completely, and the rest of this lesson is mechanics. A team that has not said it will argue about appendfsync indefinitely.
Load time is measurable in advance
Starting a Redis on a copy of the current persistence files and timing it gives the load time exactly. An RDB loads at hundreds of megabytes a second; an AOF replays at the speed of command execution, which is several times slower.
Two things usually surprise people doing this. An AOF that has not been rewritten recently loads far more slowly than one that has, because the incremental portion is long, so the recovery time depends on when the last rewrite happened. And loading is single-threaded and CPU-bound on command execution, so it does not improve with a faster disk. The only lever is keeping the replay short.
A restart is not a restore
An ordinary restart loads the persistence files and comes back with the data. A restore places files from elsewhere and then does the same. They feel identical and differ in one respect that matters: the restart's files were written by this instance and are consistent with what it last served, and the restore's were written somewhere else at some other time.
Most recovery mistakes live in that gap. Restoring a file from the wrong instance, from the wrong day, or from before a schema change all produce an instance that starts cleanly and serves wrong data, and none of them is detectable from the fact that startup succeeded. Verification after a restore exists for exactly this.
There is no partial availability during a load
Redis does not accept connections until loading completes. There is no serving of the keys already loaded and no answering with an error for the rest: the port is simply not accepting until the dataset is whole.
That is a clean design and it means recovery time is a step function rather than a ramp. Clients see a refused connection, not a slow one, which is actually easier to handle correctly: a connection refusal is unambiguous and a client library will retry it, whereas a slow or partial response invites the application to treat incomplete data as complete.
The connection storm afterwards
When the instance comes back, every client reconnects at once, and a few thousand connections arriving in the same second is real work: the accept loop, the handshakes, and whatever each client does immediately on connecting. On a large deployment that is a measurable extra delay at the worst moment.
Worse, many applications issue a burst of cache reads on reconnect, all of which miss, so the recovery is followed immediately by maximum load on whatever sits behind Redis. Recovery time measured as "Redis is serving" understates the time until the system is healthy, sometimes by a lot, and that gap is what the second section is about.
Detection and decision dominate small recoveries
For an instance that loads in 20 seconds, the recovery is rarely 20 seconds. It is however long the monitoring takes to notice, plus however long a human takes to decide what to do, plus the load, plus the storm. The load is often the smallest term.
This is why automatic failover exists and why it is worth more than a faster load on most deployments. Sentinel or Cluster can detect and promote in seconds, with no human in the path, and the recovery point cost is the replication lag at the moment of failure. Cutting the human out of a thirty-second recovery is the difference between a blip and an incident.
What the client does while waiting
Client behaviour during an outage is part of the recovery and it is usually unconsidered. A library with no timeout blocks a request thread per call, so a Redis outage becomes thread exhaustion in the application and an outage much larger than Redis. A library with aggressive retries turns the recovery into a load spike.
The settings worth deciding deliberately are the connect timeout, the command timeout, and the retry policy with backoff and jitter. Those three are what determine whether a Redis recovery looks like degraded latency or a cascading failure, and they live in the application rather than in Redis, which is why they are so often nobody's responsibility.
Warm standby changes the arithmetic entirely
A replica that is already loaded and following the stream has no load time at all. Promoting it is a role change, measured in the time to detect and decide, and the dataset is already in memory.
That is the single largest improvement available to recovery time and it is usually already paid for, because the replica exists for availability anyway. An instance with a replica and no automatic failover is leaving most of that value on the table: the expensive part, keeping a second copy warm, is already done, and only the promotion is missing.
When the host is gone too
A process failure is recovered by restarting a process. A host failure means provisioning a host first, and on most infrastructure that is minutes before Redis is even installed, plus whatever configuration management takes to make it match.
That term is frequently left out of a recovery estimate because it is somebody else's system, and it is often the largest one. The remedies are the ordinary ones: a standby host already provisioned, an image that boots ready, or a deployment mechanism that has been exercised recently. A recovery plan that begins "provision a host" and has never been timed is an estimate rather than a plan, and the difference shows up as an extra twenty minutes at the worst possible moment.
Measuring it properly
The honest measurement is an exercise, not a calculation. Kill the primary on a staging deployment with a representative dataset and a representative client load, and time from the kill to the application behaving normally again. Record each phase separately.
What that produces is usually uncomfortable and always useful. The common findings are that detection takes far longer than assumed, that one client library has no timeout configured, and that the load behind the cache cannot absorb a cold start. None of those is discoverable from a configuration review, and all three are cheap to fix once named.
What to write down
Three numbers and one procedure: measured load time, measured end-to-end recovery time, the loss window implied by the persistence configuration, and the sequence of steps for the chosen recovery path. All four fit on a page.
The reason to write them down rather than know them is that recoveries happen at inconvenient times to whoever is available, and the quality of a recovery is mostly determined by how little has to be worked out during it. A page that says what to do, in order, with the expected duration of each step, is worth more than any amount of configuration tuning.
Recovery time measured as the moment Redis starts serving understates the time until the system is healthy. Every client reconnects at once, many issue a burst of cache reads that all miss, and the recovery is immediately followed by maximum load on whatever sits behind Redis. For an instance that loads in twenty seconds, detection and the decision to act are usually larger terms than the load, which is why automatic failover is worth more than a faster load on most deployments.