0%
Redis Deep Dive: Encodings, Memory, and Failover
The Server Model
Data Structures Inside
Memory Management
Persistence
Replication and Cluster
Latency, Where It Comes From and How to Find It
Redis is fast enough that when it is slow, something specific is wrong, and the list of things that can be wrong is short and well understood. The whole subject is organised by one fact: commands execute on a single thread, so anything that occupies that thread delays everything else.
The queue is the whole model
Commands arrive, wait their turn, execute, and the reply is written back. Everything that matters about latency is a question of which of those four stages the time was spent in, and only one of them is under Redis's control.
Holding this model explicitly is what makes latency investigation tractable. A client reporting ten milliseconds when the server reports every command taking under a millisecond is not a contradiction: the time was spent queueing, in the network, or in the client, and knowing which of those three it was narrows the problem to a different team's territory.
Intrinsic versus induced latency
Intrinsic latency is what the machine itself imposes: scheduler delays, hypervisor steal, interrupt handling. Induced latency is what Redis or its workload creates: a slow command, a fork, an fsync, an eviction cycle.
The distinction matters because the fixes are completely different. Induced latency is a configuration or workload problem you can fix. Intrinsic latency is a hosting problem, and no amount of Redis tuning improves it. redis-cli --intrinsic-latency measures the floor directly, and running it first saves a great deal of wasted effort on a noisy virtual machine.
Why averages hide everything
Mean latency of 0.3 milliseconds is compatible with one percent of requests taking two hundred milliseconds, and the one percent is the entire user complaint. Every latency question is a percentile question.
This is especially true in Redis because the characteristic failures are rare and large: a fork stalling for a second, one KEYS call blocking for half a second, a single fsync taking four hundred milliseconds. These events barely move the mean and dominate the p99 and p999, so a dashboard showing only averages will report that everything is fine throughout the incident.
Why one slow command delays unrelated ones
A command taking two hundred milliseconds does not just make its own caller wait. Every other command that arrived during those two hundred milliseconds waits too, including trivially fast ones from unrelated features.
This head-of-line blocking is the single most important consequence of the single-threaded model, and it explains the most common confusing symptom in Redis operations: a feature that only issues GET on small strings experiences a latency spike, and the cause is a different feature's SMEMBERS on a set with two million members. The slow command and the slow report come from different places.
The sources, in rough order of how often they are the answer
Slow commands against large values. Fork for persistence. Transparent huge pages. Swap. AOF fsync stalls. The expiry and eviction cycles. Network and client-side problems. TLS. And the machine's own intrinsic latency.
Having this list memorised is most of the skill, because latency investigation is largely a process of elimination and the candidates are few. Each item has a specific signature in the metrics, and the following sections go through the measurement tools first and then the causes, so that each cause can be connected to the evidence that identifies it.
Commands whose cost is proportional to data
Most Redis commands are constant time, which is why the ones that are not deserve attention: KEYS, SMEMBERS, HGETALL, LRANGE over a long range, SORT, ZRANGE on a large interval, DEL on a huge collection.
None of these is inherently wrong, and all of them become incidents at a size nobody anticipated. A HGETALL written when the hash had twenty fields is fine; the same line when the hash has grown to two hundred thousand fields holds the thread for a long time. The dangerous property is that the code did not change, so nothing in the deployment history points at the cause.
Why KEYS is in a category of its own
KEYS scans the entire keyspace and builds the full reply before returning. On a database with ten million keys it blocks the server for seconds, and it is the single most reliable way to cause an outage with one command.
SCAN exists for this purpose and is an incremental cursor-based alternative that returns a small batch per call. The guidance is simple: KEYS has no place in any running system, and a code review that lets it through has let through a latency bomb that detonates when the keyspace grows. The same applies to FLUSHALL without ASYNC and to DEBUG commands.
Where the client contributes
Connection establishment, TLS handshakes, DNS resolution, a connection pool that is exhausted so requests wait for a connection, garbage collection in the client process, and a client library that parses replies slowly all appear to the application as Redis latency.
Pool exhaustion is the one worth naming specifically, because it produces a latency graph that looks exactly like a server problem and has nothing to do with the server. If client-measured latency is high while server-side commandstats and latencystats are clean, the pool and the client process are the first places to look rather than the last.
Where the network contributes
A round trip across a datacentre is a few hundred microseconds and across regions tens of milliseconds, which dwarfs Redis's execution time. One hundred sequential commands is one hundred round trips.
This is why pipelining matters so much and why it belongs in a latency discussion rather than only a throughput one. An operation that issues fifty commands sequentially cannot be faster than fifty round trips no matter how fast the server is, and converting it to one pipelined batch turns tens of milliseconds into one round trip plus the execution time.
Why p99 and p999 both matter
The p99 tells you about something happening to one request in a hundred, which at ten thousand requests per second is a hundred requests per second. The p999 tells you about the rare large stall, which is where forks and fsyncs live.
Looking only at p99 therefore misses exactly the class of problem that Redis is characteristically prone to, and looking only at p999 makes a noisy signal out of events nobody can act on. Both, plotted together, give the shape: a raised p99 with a normal p999 is usually a workload problem, and a normal p99 with occasional enormous p999 spikes is usually a fork, an fsync or the machine.
Why latency and throughput are different problems
A server handling a hundred thousand operations per second with a p99 of four hundred microseconds and the same server handling ten thousand with a p99 of forty milliseconds are in completely different trouble, and the second one is much harder to explain. Throughput problems are usually about the amount of work arriving, and latency problems are usually about one thing occupying the thread at the wrong moment.
Keeping them apart matters because the responses conflict. Pipelining improves throughput enormously and can make an individual operation's latency worse, since commands wait for a batch to assemble. Reducing a command's cost improves latency and may not change throughput at all if the server was never near saturation. Deciding which of the two is actually the complaint is therefore part of step one, and conflating them produces changes that improve the number nobody was measuring.
What to take from the sources
Three things. Every latency question is a question of which of the four stages the time was spent in, and only execution is Redis's own. Averages conceal the rare large stall that is Redis's characteristic failure mode, so percentiles are not optional. And head-of-line blocking means the feature reporting the latency is routinely not the feature causing it, which is why the investigation has to start from the server rather than from the complaint.
Measure the machine's intrinsic latency before tuning anything. redis-cli --intrinsic-latency reports the floor imposed by the scheduler, the hypervisor and interrupt handling, and on a noisy virtual machine that floor can be milliseconds. No Redis configuration improves it. Discovering it first turns a long tuning exercise into a short conversation about where the instance is hosted.