0%
Redis Deep Dive: Encodings, Memory, and Failover
The Server Model
Memory Management
Persistence
Replication and Cluster
Correct Use
Strings, and the Encodings Hiding Behind Them
TYPE on a Redis string always says string, and OBJECT ENCODING on the same key will say int, embstr or raw. The type is what the commands operate on; the encoding is what the memory looks like. Every data type in Redis has this split, and strings are the simplest place to understand it, because the three encodings are easy to describe and the differences between them are large.
The object header that everything shares
Every value in Redis is wrapped in a robj, a small structure holding the type, the encoding, a field used for eviction accounting, a reference count and a pointer to the actual data. It is sixteen bytes on a 64-bit build. That header is why a key holding the single character a does not cost one byte, and it is the fixed floor under every value in the system.
The eviction field is worth noting because it is reused rather than duplicated: under an LRU policy it holds a coarse timestamp of last access, and under an LFU policy it holds a decaying access counter plus a last-decrement time. One field, two meanings, chosen by the configured policy, which is exactly why OBJECT IDLETIME and OBJECT FREQ cannot both work on the same server.
int: no string at all
If a string's contents are a valid signed 64-bit integer in canonical form, Redis does not store a string. It stores the number in the pointer field of the object header itself, and sets the encoding to int. There is one allocation instead of two and no character data anywhere.
Canonical form matters and surprises people. SET k 123 gives int. SET k 0123 gives embstr, because the leading zero means the text would not round-trip. SET k " 123" gives embstr for the same reason, and a value beyond the 64-bit range gives embstr too. The practical consequence is that a counter or an identifier stored as a bare decimal number is markedly cheaper than the same number with any formatting applied to it, and that a zero-padded identifier costs more than the unpadded one in every key that holds it.
Shared integers, and the condition that disables them
Redis preallocates objects for the integers from 0 to 9999 and hands out pointers to them rather than allocating new ones, incrementing a reference count instead. A million keys all holding the number 1 therefore share one object, which is why OBJECT REFCOUNT on such a key returns an implausibly large number, often reported as the maximum because the count saturates.
The condition that disables this is specific and worth knowing. Sharing requires that the object carry no per-key state, and under an LRU or LFU maxmemory-policy the object header's eviction field is exactly that: per-key access information. So on a server configured with maxmemory and a non-random eviction policy, shared integers are not used for values, and the memory saving quietly disappears. An instance that appeared to fit comfortably in testing with no maxmemory set can therefore need measurably more memory once a policy is configured, which is a counterintuitive direction for a memory limit to push consumption.
embstr: one allocation for the header and the bytes
For a short string that is not an integer, Redis allocates the object header and the string data as one contiguous block, with the characters immediately following the header. This is the embstr encoding, and the threshold is 44 bytes of content.
The number is not arbitrary. The target was to fit the header, the string's own metadata and the content within a 64 byte allocation, which is both a common cache line size and a convenient allocator size class. Sixteen bytes of object header, plus the string header, plus a terminating null, leaves 44 bytes for content. The benefit is one allocation instead of two, one pointer dereference instead of two, and the whole value on one cache line, which for a workload of short values is a measurable throughput difference rather than a theoretical one.
raw: the general case
Beyond 44 bytes, the object header points to a separately allocated dynamic string. This is raw, and it is what any substantial value uses. Two allocations, two cache lines, and the ability to grow in place, which the embedded form cannot do.
The transition is one-way in an important sense. An embstr value that is modified becomes raw, and it does not go back. APPEND to a ten byte value produces a raw value even though the result is still short, and so does SETRANGE. The reason is that the modification path is implemented only for the growable representation, and converting back would be work with no benefit in the common case. The consequence for memory accounting is that a keyspace built by appending consists entirely of raw values regardless of their size, and may use noticeably more memory than the same data written with plain SET calls.
Reading the encoding, and when it is worth doing
OBJECT ENCODING key is the direct check and costs nothing. It is worth running in three situations. When sizing an instance from a sample, because the encodings of your actual values determine the per-key overhead far more than the value lengths do. When a memory figure does not match an estimate, since an unexpected raw on short values usually means something in the write path is appending rather than setting. And when designing a key format, because the difference between an identifier written as a bare integer and the same identifier with a prefix or padding is the difference between int and embstr, repeated across every key.
The one caution is not to over-fit. These are real savings and they are small per key; they matter at scale and they do not matter for a thousand keys. The reason to understand them is that they explain memory arithmetic that otherwise looks wrong, and the reason to act on them is that the actions, writing bare integers and avoiding gratuitous appends, cost nothing.
A Redis type is an interface and an encoding is an implementation, and every type has several. The commands never change, the complexity guarantees are stated against the type, and the memory and the constant factors come entirely from the encoding, which Redis chooses and changes on your behalf. Almost every surprising memory figure in Redis is an encoding that was promoted earlier than expected, and almost every surprising latency figure is a command whose cost is stated for one encoding and being paid on another.
Why the split exists at all
It is worth asking why Redis bothers, since a single general representation per type would be simpler. The answer is that the workloads Redis is used for are dominated by small values, and a general representation charges a fixed cost that small values cannot absorb. A pointer is eight bytes; an allocation header is another eight to sixteen depending on the allocator; a hash bucket and a chain pointer are more. On a value of four bytes those costs are the whole of the memory, and an instance holding a hundred million such values would spend most of its memory on structure.
The encodings are therefore not micro-optimisations but the thing that makes the common Redis workload affordable. The same reasoning produces the listpack and intset encodings for collections, where the saving is larger still, and it is why every threshold in the configuration file, hash-max-listpack-entries and its relatives, is a knob controlling the same trade: compactness and linear scanning below the threshold, overhead and constant-time access above it.
The upgrade is automatic and permanent
One general rule governs encodings across every type and is worth stating here because it recurs in every later lesson: Redis promotes an encoding when a value exceeds a threshold, and it never demotes. A hash that briefly held a thousand fields stays a hash table after nine hundred and ninety of them are deleted. A string that was appended to stays raw. A set that once held a non-integer member stays a hash table after that member is removed.
The reasoning is that demotion would need a check on every delete, would cost reallocation, and would allow a workload oscillating around a threshold to pay the conversion cost repeatedly. The consequence is that a value's encoding reflects the maximum it has ever been rather than what it currently is, so a memory report shows the historical peak shape of your data. The only way to recover the compact encoding is to rewrite the value from scratch into a new key, which is occasionally worth doing deliberately after a one-off event inflated a large number of values.