0%
Redis Deep Dive: Encodings, Memory, and Failover
Data Structures Inside
Memory Management
Persistence
Replication and Cluster
Correct Use
The Keyspace and the Life of a Command
Everything Redis stores lives in a keyspace, which is a dictionary from byte strings to objects. That sentence is almost the whole data model, and the interesting consequences come from the "almost": there are several keyspaces per server, keys are opaque bytes with conventions rather than structure, and every key carries overhead that nobody accounts for until the instance is larger than expected.
Sixteen dictionaries, and why you should use one
A Redis server starts with sixteen numbered databases, controlled by the databases directive, and a connection picks one with SELECT. They are separate dictionaries in the same process, which means they share everything that matters: one event loop, one memory limit, one persistence file, one replication stream and one set of statistics. The isolation is nominal. A slow command in database 3 stalls database 0, a key avalanche in database 7 evicts from everywhere, and maxmemory applies to the total.
The strong recommendation is to use database 0 only, for three reasons. Redis Cluster supports only database 0, so any use of the others is a decision to never shard. SELECT is per connection, which makes it a stateful command that interacts badly with connection pools, since a pooled connection returned without resetting its database hands the next caller a different keyspace. And the thing people reach for multiple databases to achieve, namespacing, is better done with a key prefix, which costs a few bytes and survives sharding.
Keys are bytes, and conventions do the rest
A key is an arbitrary byte string up to 512 megabytes, with no imposed structure. Every scheme you have seen, user:1000:profile, cache:v2:route:/home, queue:emails, is pure convention, understood by your code and by the human reading the output of SCAN and by nothing else. Redis does not index prefixes, does not know that a colon is a separator, and offers no way to enumerate the children of a prefix other than scanning every key and matching.
Three conventions earn their keep. A consistent separator, almost always a colon, makes patterns and tooling work. A type indicator near the front, so that cache: and session: and lock: are distinguishable at a glance, makes a memory report interpretable. And a version segment, so that a schema change writes v3 keys while v2 keys expire naturally, turns a migration into a deployment rather than an event.
What a key costs before it holds anything
This is the figure that surprises people sizing an instance. A key is not just its bytes. It is an entry in the main dictionary, which is a pointer to the key string and a pointer to the value object plus a next pointer for the hash chain. The key string is a dynamic string with its own header. The value is a robj with a type, an encoding, a reference count and a pointer. And the allocator rounds every allocation up to a size class.
In practice, a key holding a short string costs on the order of 50 to 100 bytes of overhead on top of the key name and the value. A million keys each holding a ten byte value is therefore not ten megabytes, it is closer to a hundred, and an instance sized on the data rather than the structure runs out of memory at a tenth of the expected capacity. This arithmetic is the single most common cause of an unexpected maxmemory event, and it is also the reason the memory lesson later in this course spends time on the encodings that let many logical values share one key.
The commands that tell you what is there
DBSIZE returns the key count for the current database, cheaply, because the count is maintained rather than computed. EXISTS takes multiple keys and returns how many of them exist, counting duplicates, which is a small subtlety worth knowing. TYPE returns the data type of a key as a simple string. RANDOMKEY returns one key at random, which is more useful than it sounds for sampling a keyspace you did not design.
OBJECT is the introspection command and has four subcommands. OBJECT ENCODING returns the internal representation, which is how you check whether a hash is still a listpack or has been promoted to a hash table. OBJECT REFCOUNT returns the reference count, which for shared integers is a large number because Redis shares small integer objects. OBJECT IDLETIME returns seconds since last access and requires an LRU eviction policy. OBJECT FREQ returns the access frequency counter and requires an LFU policy. Those last two are mutually exclusive, which catches people: you cannot have both idle time and frequency, because the same field in the object header stores one or the other.
KEYS, and why it is the most dangerous command in Redis
KEYS pattern returns every matching key. It does this by walking the entire keyspace, in one command, on the single thread, with no yield point. On a keyspace of ten million keys it takes hundreds of milliseconds to seconds, during which the server answers nothing else.
That makes it the command most likely to cause an outage, and it has two properties that make it especially dangerous. It is fast enough on a development instance with a thousand keys that nobody notices the problem until production. And it is the obvious thing to type when you want to find something, so it arrives through a human at a console during an incident, which is the worst possible moment. The correct responses are to use SCAN for anything programmatic, and to disable KEYS entirely in production through rename-command or an ACL rule, so that the obvious thing to type is simply unavailable.
Treat KEYS, FLUSHALL, FLUSHDB and DEBUG as commands that should not exist on a production instance, and remove them with rename-command or an ACL rule rather than relying on discipline. Each of them is a single line somebody types during an incident that makes the incident much worse, and in every case there is a safe alternative: SCAN for KEYS, and an explicit ASYNC flush run deliberately for the others.
Expiry is part of the keyspace, not part of the value
A key's time to live is stored separately, in a second dictionary mapping keys to expiry timestamps, which has several visible consequences. Only keys with an expiry occupy space there, so a keyspace of permanent keys pays nothing for the mechanism. PERSIST removes the entry and makes a key permanent. And crucially, most write commands that replace a value clear the expiry while some do not: SET without KEEPTTL clears it, SET with KEEPTTL keeps it, GETSET clears it, INCR and APPEND and HSET on an existing key keep it, and RENAME carries the source's expiry to the destination.
That inconsistency is a real source of bugs, because the behaviour is per command rather than general, and the failure mode is silent: a cache entry that was supposed to expire in an hour becomes permanent, memory grows slowly for weeks, and nothing in the logs mentions it. The defensive habit is to set the expiry in the same command as the write wherever a command allows it, using SET key value EX 3600 rather than a SET followed by an EXPIRE, which also removes the window where a crash between the two leaves a permanent key.
The keyspace as an operational object
One last framing that pays off later. The keyspace is not just where your data is, it is the thing you will be reasoning about during every capacity and latency incident, and the questions you will ask of it are always the same four: how many keys, how large are the largest, which ones are hot, and which ones have no expiry that should. None of those four has a single command that answers it exactly, which is why a later section of this lesson is about sampling techniques, and why a key naming scheme designed so that a prefix identifies a purpose is worth more than it looks. A memory report broken down by prefix answers a capacity question in a minute; the same report over keys named by an opaque hash answers nothing at all.