0%
Redis Deep Dive: Encodings, Memory, and Failover
Data Structures Inside
Memory Management
Persistence
Replication and Cluster
Correct Use
RESP, Pipelining, and the Cost of a Round Trip
Redis speaks a text protocol simple enough to implement in an afternoon and fast enough that nobody has needed to replace it in fifteen years. Knowing it is not academic. It is what lets you read a packet capture during an incident, understand why a client library behaves the way it does, write a shell script that talks to Redis with no client at all, and reason about the memory a request costs on the server before it is executed.
Five type markers and a line terminator
RESP, the Redis Serialisation Protocol, encodes a value as one byte saying what it is, then the value, then a carriage return and a line feed. In the original version there are five markers. A plus sign introduces a simple string, used for status replies such as +OK. A minus sign introduces an error, where the first word by convention is an error code: -WRONGTYPE Operation against a key holding the wrong kind of value. A colon introduces an integer: :42. A dollar sign introduces a bulk string, which is a length followed by that many raw bytes: $5\r\nhello\r\n. An asterisk introduces an array, which is a count followed by that many elements, each with its own marker.
The length prefix on a bulk string is the whole reason this protocol is fast despite being text. A parser never scans for a delimiter inside a value, never escapes anything, and never cares what the bytes are, so a value may contain newlines, null bytes or a JPEG. It reads a number, then reads that many bytes. Two absences have their own encodings: a null bulk string is $-1\r\n and a null array is *-1\r\n, which is how a missing key is distinguished from an empty string.
Every request is an array of bulk strings
Replies use the whole type system; requests use exactly one shape. A command is an array whose elements are bulk strings, the first being the command name and the rest the arguments. SET name mark goes out as:
That uniformity is why the server's parser is small and why argument values need no quoting rules. It also explains a cost that surprises people: every argument carries a length header of a few bytes, so a command with a thousand tiny arguments spends a noticeable fraction of its bytes on framing. For small values the framing can exceed the payload, which is one reason the wire cost of a workload is often double what a naive sum of value sizes predicts.
Reading a capture without a decoder
The practical payoff of knowing the five markers is that a packet capture becomes readable with no tooling. Filter on the Redis port, follow one stream, and the text is right there: a line beginning with an asterisk opens a command, the command name is the first bulk string, and the reply follows with its own marker. Three things are worth looking for specifically when you do this during an incident.
The first is the command mix, which is often not what the application's authors believe it is. An object-mapping layer or a caching decorator can turn one logical read into a HGETALL of a hundred-field hash, or into a KEYS call somebody added for a migration and never removed, and the capture shows it without needing anybody to agree about what the code does. The second is the reply sizes, since the bulk-string length prefixes are plainly visible and a long sequence of large lengths is an application fetching whole collections where it needs one field. The third is the interleaving, which tells you whether the client is pipelining: a run of several commands with no replies between them is a pipeline, while strict alternation of command and reply is a sequential client paying a round trip per operation.
What the protocol cannot express
Two limits follow from the length prefix, and both shape how applications are written. Because a bulk string declares its size before its bytes, a client cannot start sending a value whose length it does not yet know, so there is no streaming write: an upload must be buffered in the client's memory in full before the first byte goes out. And because a reply is similarly framed, there is no streaming read of a single large value either, which is why fetching a fifty megabyte string costs fifty megabytes of client memory and fifty megabytes of server output buffer at the same moment.
This is the real argument against using Redis as a blob store, and it is a protocol argument rather than a performance one. The commands that chunk a value, GETRANGE and SETRANGE, exist precisely as the workaround, and an application that genuinely needs to move large objects through Redis should use them deliberately rather than discovering the memory behaviour under load.
The inline form, and why it exists
There is a second request form. If the first byte of a line is not an asterisk, the server treats the line as an inline command: whitespace-separated words, parsed as the command and its arguments. This is what makes telnet host 6379 followed by PING work, and it is why debugging a connectivity problem needs no tools beyond nc. It is strictly less capable, since a value containing a space or a newline cannot be expressed, and no client library uses it for real traffic. Keep it in mind as a diagnostic: if you can reach the port and type PING and get +PONG, the problem is above the network.
What the server does with the bytes
Incoming bytes land in a per-client query buffer, and the server parses whatever complete commands that buffer now holds. Two limits live here, and both appear in incident reports. The query buffer has a hard ceiling of one gigabyte per client, beyond which the client is closed, which is the backstop against a client that opens a connection and streams nonsense. And proto-max-bulk-len, default 512 megabytes, caps the declared length of a single bulk string, which is the real limit on how large one value can be.
The argument memory is accounted separately and visibly. CLIENT LIST and CLIENT INFO report qbuf for the query buffer size, qbuf-free for its spare capacity, argv-mem for the memory held by the parsed arguments and multi-mem for queued transaction commands. A client sending an enormous MSET is visible in those fields before the command has executed at all, which makes them the right place to look when memory rises with no corresponding growth in the keyspace.
The length prefix is the whole design. Because every bulk string declares its size, Redis never scans for a delimiter inside a value and never escapes anything, so values are arbitrary bytes and the parser is a few hundred lines. The price is that a client must know a value's length before sending it, which is why streaming an unknown-length value to Redis is not something the protocol can express.
Why the protocol is not the bottleneck, and what is
It is tempting to assume a text protocol must be slower than a binary one, and for Redis the assumption is wrong for a measurable reason. Parsing a command is a handful of memchr and integer conversions on bytes already in cache, which costs a few hundred nanoseconds. A network round trip on the same host costs tens of microseconds, and across a data centre a few hundred. The protocol is therefore two to three orders of magnitude cheaper than the thing it travels over, and optimising it would move nothing.
That ratio is the single most useful fact in this lesson, because it tells you where the time goes and therefore what to fix. A client doing ten thousand sequential GET calls across a network with a 200 microsecond round trip spends two seconds, of which the server spends perhaps five milliseconds executing. The remaining 99.7 percent is waiting. No amount of server tuning touches that figure, and no faster protocol would either. The only thing that helps is sending more than one command per round trip, which is the subject of the rest of this lesson.