0%
Redis Deep Dive: Encodings, Memory, and Failover
The Server Model
Data Structures Inside
Memory Management
Persistence
Replication and Cluster
Capacity Planning and Monitoring
Sizing Redis is mostly a memory question, and the memory question is mostly about the things that are not the data. The dataset is the part everybody estimates and rarely the part that causes the incident.
What actually consumes memory
The values themselves, the keys, the per-object overhead, the hash table holding the keyspace, the expiry table, client input and output buffers, the replication backlog, replica output buffers, Lua and Functions state, and the allocator's own bookkeeping.
Most sizing exercises account for the first item and ignore the rest, which is how an instance provisioned for eight gigabytes of data needs twelve. The items that scale with the number of keys rather than with their size matter most for small-value workloads, and the items that scale with client count matter most for workloads with many connections.
Measuring per-key cost rather than estimating it
MEMORY USAGE <key> reports the bytes a specific key and value occupy, including overhead. Writing a representative thousand keys into a scratch instance and reading used_memory before and after gives a per-key figure with everything included.
This empirical approach is far more reliable than arithmetic, because the per-object overhead depends on encodings that change with size and content in ways that are difficult to predict from a schema. Ten minutes with a scratch instance replaces an estimate that is routinely wrong by a factor of two, and the resulting number is the one to multiply by the projected key count.
Why the encoding decides the answer
A hash with sixty small fields is a listpack, which is compact. The same hash with two hundred fields is a hash table, which is several times larger. The threshold is a configuration value, and crossing it changes the memory per object abruptly.
So a sizing calculation based on average object size can be badly wrong if the distribution straddles an encoding boundary, and the growth curve has a step in it rather than being linear. This is covered in the data-structure lessons, and the capacity consequence is that the measurement above must be taken with objects of realistic size rather than with small examples.
The fragmentation gap
Redis asks the allocator for memory and the allocator asks the operating system, and the resident size is larger than the sum of the allocations. The ratio of used_memory_rss to used_memory is typically between 1.0 and 1.5.
That gap is real memory that must be provisioned, so a dataset of ten gigabytes may need fifteen gigabytes of resident space. The ratio rises after a workload that writes and deletes a lot of differently sized objects, and activedefrag can reclaim some of it at a CPU cost. Treating the ratio as part of the sizing rather than as an anomaly avoids a class of surprise.
Headroom for copy-on-write
If forking persistence is enabled, a save requires memory for every page the parent modifies while the child runs. On a write-heavy instance that can be a substantial fraction of the dataset, and it is needed on top of everything else.
The conservative guidance that a Redis host should have roughly twice the instance's memory comes from here, and it is conservative rather than wrong: the actual requirement depends on the write rate during the save and on whether transparent huge pages are disabled. Sizing with no copy-on-write headroom on an instance that forks is how a save turns into an out-of-memory kill.
Client buffers, which scale with connections
Each client has an input buffer and an output buffer. A thousand idle connections cost little; a thousand connections each receiving a large reply cost a great deal, and the client-output-buffer-limit settings exist to bound that.
The case that bites is a slow consumer: a client that reads its socket slowly while the server keeps producing output accumulates megabytes in its output buffer. Replicas have their own, larger limits for the same reason, and a replica that cannot keep up during a full synchronisation is the textbook instance of this, covered in the replication lesson.
The replication backlog
The backlog is a fixed-size circular buffer of recent writes, sized by repl-backlog-size, and it exists so that a briefly disconnected replica can resynchronise partially rather than fully. It is allocated whether or not it is needed.
Sizing it is a capacity decision with an availability consequence: too small and a short network blip forces a full synchronisation, which is expensive and happens exactly when the system is already stressed. The right size is the write rate multiplied by the longest disconnection worth surviving, which makes it a number to derive rather than to leave at the default.
CPU, which is simpler than it looks
One core executes commands and cannot be parallelised. I/O threads, if enabled, handle reading and writing sockets on additional cores. A fork for persistence uses another core for the duration of the save. Active defragmentation uses CPU when enabled.
So the practical floor is two cores and the comfortable figure is four, with more only helping if I/O threads are enabled and the workload is dominated by socket work rather than command execution. Provisioning sixteen cores for a Redis instance and expecting proportional throughput is a common and expensive misunderstanding of the threading model.
Network, which is often the real limit
An instance serving a hundred thousand operations per second with one kilobyte values moves a hundred megabytes per second, which is most of a gigabit link. Large values make this the binding constraint well before CPU or memory.
This is worth calculating explicitly, because it is the limit people forget and the one that produces the most confusing symptoms: latency that rises with throughput, no slow commands, normal CPU, normal memory. INFO stats reports total_net_input_bytes and total_net_output_bytes, and comparing their rate against the interface capacity settles the question quickly.
Projecting growth honestly
Capacity planning needs a growth rate and a horizon. The useful question is not how much memory the data needs now but how long the current instance lasts at the observed growth rate, and whether that is longer than the lead time to change it.
Stating it that way converts a vague worry into a date, and a date is actionable. An instance with forty percent headroom growing at five percent per month has about seventeen months, which is comfortable. The same instance growing at five percent per week has about two months, which needs a decision now, and the difference is invisible in a snapshot of current usage.
Why maxmemory must be set
Without maxmemory, Redis grows until the operating system refuses, and the result is an out-of-memory kill rather than eviction. Setting it is the difference between a degraded cache and a dead process.
The value should leave room for everything in this section that sits outside used_memory: fragmentation, copy-on-write, client buffers. A common arrangement is maxmemory at somewhere between half and three-quarters of the host's memory, depending on whether the instance forks and how write-heavy it is, and the exact fraction matters less than the fact that a limit exists at all.
What to take from sizing
Three things. Measure per-key cost empirically on a scratch instance rather than calculating it, because encodings make arithmetic unreliable. Budget for everything outside the dataset, since fragmentation, copy-on-write headroom, client buffers and the replication backlog together are frequently larger than the estimate they are added to. And express the result as a date rather than a number, because how long the instance lasts at the observed growth rate is the only form of the answer anybody can act on.
Express a capacity answer as a date, not a number. Forty percent headroom growing at five percent per month is seventeen months and needs nothing; the same headroom growing at five percent per week is two months and needs a decision today. Both look identical in a snapshot of current usage, which is why a dashboard showing memory used is not a capacity plan.