Every surprising thing about Redis follows from one design decision, and the decision is older than most of the features built on top of it: commands execute one at a time, on one thread, inside one loop. Nothing in the data path takes a lock, because nothing in the data path can be reached by two threads at once. Understanding the loop is therefore not background material. It is the model from which the memory behaviour, the latency behaviour, the persistence behaviour and the failover behaviour all follow.

The loop is the whole server

A running Redis server, stripped of everything optional, is a function that does this forever: ask the kernel which of several thousand file descriptors are ready, handle each one, run any timer that has come due, and ask again. In the source this is aeMain in ae.c, and the body is a call to aeProcessEvents. The name ae stands for "a simple event-driven programming library", which is the comment at the top of the file, and the library is about six hundred lines long. That is the server's scheduler.

Each pass through the loop is called a cycle. A cycle does four things in a fixed order. First it runs a beforeSleep callback, which is where Redis does the work it has deliberately deferred: flushing the append-only file buffer to the operating system, writing accumulated replies out to client sockets, propagating writes to replicas, handling clients that became unblocked, and running a short burst of key expiration. Then it calls the kernel to wait for readiness, with a timeout derived from the nearest pending timer. Then it runs an afterSleep callback. Then it dispatches every ready file descriptor and every expired timer.

The ordering matters more than it looks. Replies are not written when a command produces them. A command appends bytes to a per-client output buffer and returns, and the socket write happens later in the same cycle, in beforeSleep. This is why a pipeline of a thousand commands can be answered with a handful of write calls rather than a thousand, and it is the main reason pipelining is as effective as it is.

What the kernel is asked, and what it answers

The readiness call is whichever multiplexing interface the platform offers, chosen at compile time rather than at runtime. On Linux it is epoll. On macOS and the BSDs it is kqueue. On Solaris and derivatives it is evport. If none of those exist it falls back to select, which is where the old 1024 descriptor ceiling came from. The preference order is in a short block of conditional includes, and this is one of the rare places where the behaviour of a Redis server genuinely depends on which operating system built it.

What comes back is a list of descriptors that will not block if read or written. It is not a list of complete requests. A readable socket may hold a fragment of a command, three complete commands, or two and a half. Redis reads up to a configurable amount into a per-client query buffer and then parses whatever complete commands that buffer now contains, leaving the remainder for the next time the socket is readable. A client that sends *3\r\n$3\r\nSET and stops talking leaves a partial command sitting in a buffer, consuming memory and nothing else.

A command is executed between two polls

The path from bytes to effect is short and worth memorising, because every performance question in Redis is a question about where in this path the time went. readQueryFromClient fills the query buffer. processInputBuffer parses complete commands out of it. processCommand looks the command up in a table, checks arity, checks authentication and ACLs, checks whether the server is accepting writes at all, and then calls call(). call() runs the command's implementation, which touches the keyspace directly and appends a reply with addReply. call() also records the command's duration, propagates it to the append-only file and to replicas if it changed anything, and updates the per-command statistics.

Nothing in that path yields. There is no point at which a half-finished command is parked so that another client can be served. A command that takes four hundred microseconds occupies the server for four hundred microseconds, and a command that takes four hundred milliseconds occupies the server for four hundred milliseconds.

Time events, and the job that runs ten times a second

Alongside file descriptors the loop holds time events, and in practice there is one that matters: serverCron. It runs at a frequency set by the hz directive, which defaults to 10, so ten times a second, and dynamic-hz yes raises that frequency when the server has many clients so that per-client housekeeping keeps up.

serverCron is where everything periodic lives. It runs the slow cycle of active key expiration. It advances incremental rehashing on dictionaries that are growing or shrinking. It walks a slice of the client list looking for idle connections to close and oversized query buffers to shrink. It samples memory usage and updates the clock that approximated LRU reads. It decides whether a background save or an append-only file rewrite is now due. It drives the cluster bus bookkeeping. None of that work is free, and all of it happens on the same thread as your commands, which is the first hint that hz is a latency setting as much as a responsiveness setting.

What atomic means here, and what it does not

Because commands run one at a time to completion, every single command is atomic with respect to every other command. INCR cannot interleave with another INCR. LPUSH of ten elements is not visible as six elements. A Lua script runs as one unit for the same reason, and so does the body of a MULTI and EXEC block: they are executed back to back with nothing in between.

What this does not give you is atomicity across round trips. A client that runs GET balance, decides in its own process, and then runs SET balance, has performed two atomic operations with an unbounded gap between them, and any number of other clients may have acted inside that gap. The single-threaded server makes each step indivisible and says nothing at all about the sequence. That distinction is the entire reason WATCH, Lua scripts and the Functions API exist, and it is the most commonly misread consequence of the threading model.

Why this design was chosen, and what it cost

It is worth being explicit that the single-threaded loop was a choice rather than an omission, and that the reasoning has aged well. A key-value store whose values are in memory spends almost no time per operation: a hash lookup and a pointer dereference, measured in tens of nanoseconds. Against that, the cost of acquiring and releasing a lock is of the same order as the work being protected, and the cost of a cache line bouncing between cores is larger. A multithreaded design with fine-grained locking would therefore have spent a large share of its budget on coordination, and a design with one coarse lock would have been a single-threaded server with extra steps and worse latency.

The choice also bought simplicity that users feel directly. Every command is atomic with no annotation. There are no isolation levels to learn, no deadlocks to detect, no lock ordering rules to get wrong, and no partial visibility of a multi-element write. Features that are hard in concurrent systems, such as MULTI and EXEC, Lua scripting and keyspace notifications, were cheap here because the serialisation they need already existed. The source stayed small enough that a reader can follow a command from socket to effect in an afternoon, which is not a small factor in how reliable Redis has been.

What it cost is a ceiling of one core for execution, and the consequence that a single expensive operation is a server-wide pause. The project's answer to the first has consistently been horizontal: run more instances, shard the keyspace, and use the cores that way. That answer is also why Redis Cluster exists in the shape it does, with a slot map and client-side routing rather than a coordinating layer, because the unit being multiplied is a whole single-threaded server. The answer to the second is discipline in what you ask for, which is the subject of the third section of this lesson.

"Redis is single threaded" is a sentence that was precisely true in 2010 and has been getting less true, in careful increments, ever since. A modern Redis process has several threads, a background child process at times, and in recent versions a thread pool that reads and writes sockets. What has never changed is the part that matters: the keyspace is touched by one thread. Every extension to the threading model has been designed to preserve that invariant, which is why none of them introduced locks into the data structures.

What the background threads do

The oldest extension is bio.c, the background I/O subsystem, which exists because some operations are slow for reasons that have nothing to do with the data structures and everything to do with the kernel. Closing a large file can block. Calling fsync can block for a long time under disk pressure. Freeing a ten million element hash is pure CPU work on memory the main thread no longer needs anybody to see. These are handed to background threads through small job queues.

The classic three job types are closing a file descriptor, fsyncing the append-only file, and lazy freeing. Lazy freeing is the one with user-visible configuration. UNLINK is DEL with the free handed off, and five directives control whether deletions caused by the server rather than the user also get handed off: lazyfree-lazy-eviction, lazyfree-lazy-expire, lazyfree-lazy-server-del, lazyfree-lazy-user-del and lazyfree-lazy-user-flush. All of them default to no, which is worth knowing because the default therefore frees a very large evicted object on the loop thread. The handoff is only worth the bookkeeping above a size threshold, so a small object is freed inline even when lazy freeing is enabled, and an object with more than one reference cannot be freed in the background at all.

What the I/O threads do

Redis 6.0 added a thread pool for socket work, configured with io-threads. The count includes the main thread, so io-threads 4 means three additional threads. By default it is 1, which means no additional threads at all, and this remains the default in the stock configuration file. In the 6.x and 7.x line writes were offloaded by default when the pool was enabled, and reads only when io-threads-do-reads yes was also set. The Redis 8 line reworked this part substantially, folding the reads directive in and making the pool handle both directions with additional work such as prefetching keys before execution, so if you are reading a configuration file written for an older version, expect that directive to be either deprecated or absent.

What the pool does not do, in any version, is execute commands. The main thread still parses and executes every command against the keyspace, one at a time, in a single sequence. The threads move bytes between sockets and per-client buffers, which is work the main thread was previously doing with syscalls, and on a workload dominated by small values and many connections that work is a large fraction of the total. That is the whole of the benefit and also the whole of the limit: enabling I/O threads can raise throughput substantially on a machine with cores to spare, and it does nothing for a workload whose bottleneck is a command that takes a long time to execute.

What the fork child does

The third form of concurrency is a separate process rather than a thread. BGSAVE, BGREWRITEAOF and a full synchronisation to a replica all begin with fork. The child inherits a copy-on-write view of the parent's memory, walks the keyspace it sees at the instant of the fork, and writes a file. The parent keeps serving commands.

Two consequences deserve stating now and get a whole section later. The fork itself is a pause on the main thread, proportional not to the data but to the size of the page tables being copied, and latest_fork_usec in INFO reports the last one. And every page the parent writes to while the child lives must be copied, so a write-heavy server during a save can use far more memory than its dataset, which is the single most common cause of an unexpected out-of-memory kill on a Redis host.

Modules, and the one place a module may use the keyspace off-thread

The module API adds threads of its own. A module may start a thread and block a client while that thread works, then reply when it is done, which is how a module can perform a long computation without holding the loop. The constraint is that such a thread may not touch the keyspace freely; it must take the global lock, which serialises it against the main thread exactly as if it were running on it. The invariant survives: one thread at a time in the data structures, whoever that thread belongs to.

How to read the thread list on a live server

On a running Linux host, ps -T -p <pid> or looking at /proc/<pid>/task shows what is actually there. You should expect to see the main thread, three or more bio threads, the jemalloc background purge thread, and one thread per configured I/O thread beyond the first. Seeing seven or eight threads on a server you believed to be single threaded is not a sign of anything wrong. The question to ask is which of them is at a hundred percent of a core, and in almost every real deployment the answer is the main one.

What this means for how you scale Redis

Put the three mechanisms side by side and a scaling rule falls out. Background threads remove kernel-bound work from the loop, so they help a server stalling on fsync or on freeing large objects. The I/O pool removes byte movement from the loop, so it helps a server whose time is going into syscalls for many small operations. The fork removes snapshot writing from the loop, so it helps a server that must persist. None of the three removes command execution from the loop, and therefore none of them raises the ceiling on how much work the commands themselves can do.

That ceiling is one core, and the only way past it is more Redis processes. On a single host this is as simple as running several instances on different ports with the keyspace divided between them, which many teams do and which trades a little operational complexity for a near-linear increase in capacity. Across hosts it is Redis Cluster, where sixteen thousand three hundred and eighty four hash slots are distributed over the primaries and the client routes each key to the right one. The shape of the cluster design makes much more sense once you see that the unit being replicated is a whole single-threaded server with its own loop, rather than a thread in a shared pool.

The practical consequence for capacity planning is that the figure to watch is not CPU percentage across the machine but the saturation of one thread. A sixteen core host running one Redis instance is at its limit when one core is busy, and the other fifteen cores say nothing useful about how much headroom remains. This is the single most common mistake in Redis capacity dashboards: a graph of host CPU at six percent on a server that is already dropping requests, because the six percent is one saturated core averaged across sixteen.

The loop gives you atomicity for free and charges for it in a currency called head-of-line blocking. Everything waiting behind a slow command waits for the whole of it, and because Redis is usually asked for sub-millisecond answers, the ratio between a normal command and a slow one is enormous. This section is the arithmetic of that, because the arithmetic is what makes the rule memorable.

The arithmetic of one slow command

Take a server handling 50,000 operations per second, which is unremarkable. That is one command every 20 microseconds on average. Now run a single KEYS * against a million keys, which takes on the order of 300 milliseconds. During that command the server accepts no requests, so by the time it finishes, roughly 15,000 commands have piled up in socket buffers and query buffers behind it. Every one of those clients observed a latency of up to 300 milliseconds, on a service whose normal response is a quarter of a millisecond.

The effect does not stop when the command ends. The server now has a backlog to drain, during which arrival continues, so the queue takes longer than the command did to disappear. This is the standard queueing result: at high utilisation, waiting time grows much faster than service time, and one long service event produces a latency tail that outlives it. In a dashboard this appears as a one-second spike in p99 with no corresponding spike in anything that looks like a cause, because the cause was one command that ran once.

The commands that do this

The population of dangerous commands is small and learnable, and it is defined by complexity in the size of the data rather than in the size of the request.

KEYS walks the entire keyspace and should be treated as a command that does not exist in production; SCAN is its cursor-based replacement, which returns a bounded number of elements per call and spreads the same work across many loop cycles. The same relationship holds for HSCAN, SSCAN and ZSCAN against their all-at-once counterparts. SMEMBERS on a set with millions of members, HGETALL on a hash with a hundred thousand fields, LRANGE mylist 0 -1, and a full ZRANGE all build a reply proportional to the collection, which costs both the traversal and the memory for the reply buffer. SORT can be worse than all of them, since it may sort a large collection and apply patterns per element. SUNIONSTORE and SINTERSTORE across large sets are proportional to the inputs. FLUSHALL and FLUSHDB without the ASYNC argument free the entire keyspace inline.

Deletion is the trap with the sharpest edge, because DEL looks cheap. DEL on a key holding one string is one free. DEL on a sorted set with ten million members frees ten million allocations, which is hundreds of milliseconds of pure allocator work on the loop thread. UNLINK exists precisely for this: it unlinks the key immediately and hands the freeing to a background thread, so the loop pays only the unlink. On a large collection the difference is the difference between an invisible operation and an incident.

The slow things that are not commands

Not every stall has a command to blame, and the remainder are the ones that waste the most investigation time.

The fork for a background save pauses the main thread while the kernel copies page tables. On a large instance this can be tens or hundreds of milliseconds, and latest_fork_usec records it. Mass expiration can bite: if a million keys share one expiry second, the active expiration cycle has a great deal to do at once, and the cycle is bounded by an effort limit precisely so that it does not monopolise the loop, which means the deletions spread across cycles instead. Eviction under maxmemory pressure runs inline, sampling and deleting keys in the middle of serving commands, and with a large object that means an inline free as well unless lazy freeing is on. Writing to the append-only file happens in beforeSleep, and with appendfsync always the fsync is on the loop, which is why that setting costs what it costs. Swap is the worst of all: if any part of the dataset has been paged out, a command that touches it waits on disk while holding the loop, and a Redis instance that is swapping shows latency numbers that no amount of command tuning explains.

The policy that prevents it rather than the fix that follows it

Finding the slow command after the incident is the easy half. The harder and more valuable half is a standing policy, and three rules cover nearly all of it.

First, treat every command's complexity as part of the review. Redis documents the complexity of every command, and a hot path should contain only commands whose cost is independent of collection size or bounded by a count the caller supplies. LRANGE key 0 9 is fine forever; LRANGE key 0 -1 is a time bomb whose fuse is however long it takes the list to grow. The question to ask of any new call is not whether it is fast today but what happens when the collection behind it is a hundred times larger, since nothing in Redis will warn you when it is.

Second, make the keyspace-shaped operations cursor-based by default. SCAN, HSCAN, SSCAN and ZSCAN all return a bounded slice per call and spread their work over many cycles, and their guarantee is weaker than their all-at-once counterparts in a way that rarely matters: elements present for the whole iteration are returned at least once, elements added or removed during it may or may not appear. Most code that uses KEYS is doing maintenance where that guarantee is plenty.

Third, decide the deletion policy once and encode it. Anything that might hold more than a few thousand elements should be removed with UNLINK rather than DEL, and the lazyfree directives should be set to yes on any server where eviction or expiry can reach a large object, since all five default to no. These are three configuration lines and one search-and-replace, and they remove an entire category of incident.

How to find the one that is hurting you

Four tools, in the order they are usually useful. The slowlog, enabled by default at slowlog-log-slower-than 10000 microseconds, records commands whose execution exceeded the threshold, with their arguments and the client that sent them. The latency monitor, off by default until latency-monitor-threshold is set to a non-zero number of milliseconds, records events by class, including command, fork, expire-cycle, eviction-del and aof-write, which is how you distinguish a slow command from a slow fork. INFO commandstats and INFO latencystats give per-command call counts, total time and percentiles, which is how you find a command that is not individually slow enough for the slowlog but is called often enough to dominate. And redis-cli --latency and its --latency-history and --latency-dist variants measure what a client actually experiences, which includes the queueing the other three do not show.

That last point is the one people miss. The slowlog measures execution time only. It excludes the time a command spent waiting in a socket buffer behind somebody else's KEYS. So a server can have an empty slowlog and terrible client-observed latency, and the gap between those two measurements is exactly the queueing this section is about.

Redis offers commands that wait: BLPOP waits for an element, XREAD BLOCK waits for a stream entry, WAIT waits for replica acknowledgements. On a single-threaded server these look impossible. If a command runs to completion before the next one starts, how can a command wait five minutes without stopping everything? The answer is a specific and elegant mechanism, and understanding it is the difference between using these commands confidently and avoiding them out of suspicion.

A blocked client is not a blocked command

The command does not wait. It returns, having arranged for the client to be reconsidered later. BLPOP checks whether the list has an element; if it does, the command behaves exactly like LPOP and returns the element. If it does not, the command marks that client as blocked, records which keys it is blocked on and what to do when one of them is ready, sets a timeout, and returns from the command implementation. The loop continues immediately. No reply has been sent to that client, so the client sits waiting on its socket, but the server is not waiting for anything.

The server keeps a dictionary from key to the list of clients blocked on that key, which is how the wake-up is cheap. When any command makes a key ready, by pushing to the list, adding to the sorted set or appending to the stream, it calls a signalling function that adds the key to a ready list. Then, at a defined point in the cycle, the server walks the ready list and serves the blocked clients for each ready key, in the order they blocked. That ordering is the useful guarantee: among several clients blocked on the same key, the one that has waited longest is served first, so BLPOP across many workers is a fair queue rather than a lottery.

Timeouts, and the cost of a blocked client

Every blocking command takes a timeout, and zero means forever. Timeouts are not polled per client; the server keeps blocked clients in a structure ordered by deadline so that finding the ones that have expired is cheap however many there are. An expired blocking command returns the type's empty reply, which for BLPOP is a null, and that is the signal for a worker loop to call it again.

The cost of a blocked client is a connection, the memory for its structures, and nothing else. It consumes no loop time while it waits. This is why blocking commands are the right way to build a worker that waits for work, and polling with LPOP in a loop is the wrong way: a thousand workers polling at ten hertz is ten thousand commands per second of pure overhead, while a thousand workers blocked on BLPOP is a thousand idle connections and zero commands. blocked_clients in INFO clients reports how many there are, and on a healthy queue-shaped workload that number should be roughly the size of your worker pool.

Where blocking quietly does not block

There are two contexts in which blocking commands cannot block, and both exist for the same reason: there is no way to park the caller.

Inside MULTI and EXEC, a blocking command behaves as its non-blocking equivalent. BLPOP inside a transaction on an empty list returns a null immediately rather than waiting, because the transaction must execute as one indivisible unit and cannot be suspended halfway through. The same is true inside a Lua script or a Function: the script runs to completion on the loop, so a blocking command inside it cannot wait. This is the correct behaviour and it is also a very common bug, because the code reads as though it waits and does not.

There is one exception worth knowing: the module API provides a way for a module command to block a client on keys, which is how a module can offer a genuinely blocking primitive. That is the module getting access to the same parking mechanism the built-in commands use, not an exception to the model.

What a blocked client does not protect you from

One property of the parking mechanism is easy to misread as a stronger guarantee than it is. When a key becomes ready, the server serves the clients blocked on it, but the element they receive is taken at the moment they are served, not at the moment the key was signalled. If a non-blocking LPOP from another client arrives in the same cycle and runs first, the blocked client may find the list empty again and go back to waiting. This is correct and it is also why a blocking read is not a reservation: nothing is held for a blocked client until it is actually served.

Replication makes this sharper. A blocking pop that succeeds is propagated to replicas as the effect it had, which is an LPOP, so a replica never replays the waiting. But a client blocked on a primary that is failed over is blocked on a server that is no longer the primary, and when the connection is re-established against the new primary the block is simply gone; the client sees a closed connection or a MOVED redirection depending on the topology. A worker that assumes a blocking call will either return work or time out, and never simply fail, will stall at exactly the wrong moment, which is another reason the loop around it needs to treat errors as a normal outcome.

The other things that block, and the ones to reach for

Beyond the list and stream commands, WAIT blocks the client until a given number of replicas have acknowledged the writes it has issued so far, or a timeout passes, and it returns the number that did. WAITAOF, added in 7.2, does the analogous thing for fsync on the local server and on replicas, which is the stronger guarantee of the two since an acknowledgement is not a durable write. Both are per-client and per-connection: they say something about the writes that connection sent, not about the server in general.

CLIENT PAUSE deserves a mention because it is the deliberate version of the thing this whole lesson warns about. It stops the server from processing commands from clients for a specified number of milliseconds, either all commands or only writes, and it exists to create a quiet window for a failover. CLIENT UNPAUSE ends it early. Used carelessly it is an outage with a configuration flag.

The shape of a correct worker loop

Given all of this, the worker loop that behaves well under every condition has a specific shape, and each part of it corresponds to something in this section.

The timeout should be non-zero. A worker blocked with a timeout of zero never gets control back, which means it cannot check a shutdown flag, cannot emit a heartbeat, and cannot notice that the connection it is blocked on has silently died somewhere in the network. A timeout of a few seconds costs one command per worker per timeout period, which is nothing, and turns the loop into something that can be shut down cleanly and monitored honestly. The null reply on timeout is not an error condition and should not be logged as one.

The connection should be dedicated. A blocked connection cannot be used for anything else, so a worker that shares one connection between its blocking read and its other work will serialise the two in a way that is confusing to debug. Most client libraries handle this by giving the blocking call its own connection from the pool for the duration, but it is worth checking rather than assuming, because a pool that hands out a blocked connection to another caller produces symptoms that look like server faults.

The pool size should be chosen against maxclients and the file descriptor limit rather than against the work. Two hundred workers is two hundred connections sitting idle, which is well within the default of ten thousand, but five thousand workers across several services is not, and the server lowers maxclients silently at startup if the process limit cannot support it. The figure to watch is blocked_clients, which on a healthy queue should be approximately the number of idle workers, and a value well below the pool size means workers are busy rather than waiting, which is the signal that the pool is the bottleneck.

The loop has a small number of settings and a surprisingly rich set of measurements, and the measurements are the half that gets skipped. This section covers what you can turn and what you can read, with the specific field names, because an incident is not the moment to be discovering which INFO section holds the number you want.

The settings that touch the loop directly

io-threads sets the size of the socket I/O pool, counting the main thread, and defaults to 1. On the 6.x and 7.x line io-threads-do-reads controls whether reads are offloaded as well as writes, defaulting to no; in the 8 line the threading was reworked and that directive was folded in. hz sets the frequency of serverCron and defaults to 10, with dynamic-hz yes raising it under high client counts. Raising hz makes expiration, client timeout handling and buffer trimming more prompt and gives the loop more housekeeping to do, so values far above the default are a trade rather than an improvement.

maxclients defaults to 10000 and is not purely advisory: the server also needs file descriptors for replicas, the cluster bus, the append-only file and internal use, so it reserves a margin and will lower maxclients at startup with a log line if the process file descriptor limit cannot support the configured value. That log line is worth searching for on any host where connection counts are high, because a silently reduced maxclients presents as unexplained connection errors.

tcp-backlog defaults to 511 and bounds the kernel's accept queue, which matters during a connection storm, when many clients reconnect at once after a failover. timeout defaults to 0, meaning idle clients are never closed, and tcp-keepalive defaults to 300 seconds, which is what detects a peer that vanished without closing. maxmemory-clients, added in 7.0, caps the memory used by client buffers and allows the server to evict clients rather than die when a few of them accumulate enormous replies. busy-reply-threshold, previously lua-time-limit, defaults to 5000 milliseconds and is the point after which a long-running script makes the server start replying BUSY to other clients rather than appearing hung; note that it does not stop the script, it only changes what everybody else is told.

The fields that describe the loop

INFO stats holds the rate figures. instantaneous_ops_per_sec is the sampled command rate. total_net_input_bytes and total_net_output_bytes give the traffic, and the ratio of output bytes to commands is a quick way to spot a workload returning much more data than anybody intended. expired_keys and evicted_keys grow when expiration and eviction are doing work, and a step change in either at the moment of a latency spike is a strong attribution.

From 7.0 there are fields that measure the loop itself, which is the direct measurement people spent years approximating: eventloop_cycles and eventloop_duration_sum as cumulative counters, plus instantaneous_eventloop_cycles_per_sec and instantaneous_eventloop_duration_usec. Dividing duration by cycles gives the average time the server spends busy per cycle, and that number against the cycle rate tells you how close the loop is to saturated. A loop spending 900 microseconds busy per cycle at 1,000 cycles per second has no headroom left, whatever the CPU graph says.

INFO clients gives connected_clients, blocked_clients, cluster_connections and the watching and tracking counts. INFO cpu gives used_cpu_sys and used_cpu_user for the process and separate figures for the children, and the main thread's ceiling is one core, so a server at one core of combined user and system time is at its limit and will not go faster for any reason other than doing less work.

The per-command and per-event instruments

INFO commandstats gives, per command, the call count, the total microseconds, the average, the rejected count and the failed count. This finds the command that is not slow enough for the slowlog but is called often enough to own the server: a command averaging 300 microseconds called 20,000 times a second is using 6 seconds of CPU per second of wall clock, which is impossible, and seeing that arithmetic fail is how you discover you are looking at the wrong instance or the wrong window. INFO latencystats, added in 7.0, gives per-command latency percentiles directly, which is more useful than an average because the distribution is what clients feel.

SLOWLOG GET returns recent commands whose execution exceeded slowlog-log-slower-than, default 10000 microseconds, keeping slowlog-max-len entries, default 128, with the arguments and the client address. Remember what it excludes: queueing time, network time, and the time to write a large reply. LATENCY LATEST and LATENCY HISTORY require latency-monitor-threshold to be set and then attribute spikes to named classes, among them command, fork, expire-cycle, eviction-del, eviction-cycle, aof-write and aof-fsync-always. LATENCY DOCTOR reads the same data back in prose and is a reasonable first command in an incident.

Measuring from outside, and the one measurement people forget

redis-cli --latency issues PING repeatedly and reports minimum, maximum and average in milliseconds; --latency-history prints a rolling series; --latency-dist draws a distribution. All three measure what a client experiences, including the queueing that the server-side instruments omit, so running one from the application's own network position is the measurement that settles arguments about whether the server or the network is at fault.

The forgotten one is redis-cli --intrinsic-latency <seconds>, which must be run on the server host itself. It measures the latency of the machine rather than of Redis, by timing a tight loop and reporting the worst pause, and it answers a question nothing else does: how much of the observed jitter is the host, the hypervisor or the kernel, before Redis is involved at all. On a noisy virtual machine the intrinsic latency can be milliseconds, and on such a host no amount of Redis tuning will produce sub-millisecond p99, which is worth knowing before a week is spent trying.

Reading these together in an incident

Under pressure the useful thing is not a list of instruments but an order to use them in, and the order follows from what each one excludes.

Start with LATENCY DOCTOR, because it reads the latency monitor's history back in prose and will frequently name the cause in one command, and because it costs nothing to try. If the monitor was never enabled, set latency-monitor-threshold to 100 immediately so that the next occurrence is recorded, then continue without it. Next read INFO once in full and keep the output, because several of the fields you will want are cumulative counters whose value lies in the difference between two readings a minute apart: eventloop_duration_sum, eventloop_cycles, expired_keys, evicted_keys, total_net_output_bytes and rdb_changes_since_last_save all say far more as a rate than as an absolute.

Then split the question in two. If busy time per cycle is high, the server is doing too much work and the next step is commandstats and latencystats to find out which work. If busy time per cycle is low but clients are slow, the server is stalling rather than labouring, and the next step is the fork, expiry, eviction and fsync figures, with latest_fork_usec and the latency monitor classes being the fastest discriminators. Those two branches have almost no overlap in their remedies, which is why establishing which one you are in before investigating saves more time than any individual instrument.

Keep one honest caveat in view throughout: every server-side measurement excludes the queue. A server can be entirely healthy by its own account while clients see hundreds of milliseconds, because the time was spent waiting in a socket buffer behind something the server considered perfectly ordinary. Running redis-cli --latency-history from the application's own network position is the measurement that closes that gap, and it is the one most often skipped because it has to be run from somewhere other than where you are already logged in.