When a Cluster Loses a Node, and What It Cannot Promise

Topics Covered

Failure Detection in a Cluster

Every node pings every other node

PFAIL, a private suspicion

FAIL, the collective verdict

Only primaries vote

cluster-node-timeout, the one knob

What a sensible value is

What makes a node look failed when it is not

The bus port and the handshake

What CLUSTER NODES tells you

How this differs from Sentinel

What to take from detection

A cluster has no Sentinel. The primaries themselves detect failures and authorise promotions, using the gossip on the cluster bus, and one configuration setting dominates the whole process.

Every node pings every other node

Each node sends a ping to a few randomly chosen nodes every second, and additionally pings any node it has not heard from in half the node timeout. Over time that produces a full mesh of recent contact without every node pinging every other node every second.

The traffic is therefore proportional to the square of the node count in the limit, which is one of the reasons the cluster size is capped at 1000 nodes. At a few dozen nodes it is negligible. At several hundred it is a real amount of bus traffic, which is a consideration when choosing how many shards to run and another argument for shards in the tens of gigabytes rather than the single digits.

PFAIL, a private suspicion

If a node gets no reply from another for longer than cluster-node-timeout, it flags that node PFAIL, meaning possibly failing. This is one node's private opinion and causes nothing to happen.

PFAIL is the direct analogue of Sentinel's subjectively down state, and it exists for the same reason: a node that cannot reach another may be the isolated one. The flag is visible in CLUSTER NODES output, and a node showing PFAIL from one observer and not from others is the clearest possible sign of a network problem between a specific pair rather than a failure of either.

FAIL, the collective verdict

Nodes gossip their PFAIL flags to each other. When a primary learns that a majority of primaries have flagged the same node PFAIL within the node timeout window, it promotes the flag to FAIL and broadcasts that, and every node accepts it.

So the threshold is a majority of the primaries that own slots, which is not configurable and is the cluster equivalent of Sentinel's election majority rather than of its quorum. Cluster collapses Sentinel's two numbers into one, which removes the commonest Sentinel misconfiguration entirely but also removes the ability to tune detection separately from authorisation.

Cluster collapses Sentinel's two numbers into one. There is no tunable quorum, only a fixed majority of slot-owning primaries, which removes a misconfiguration and removes a tuning knob.

Only primaries vote

Replicas participate in the gossip and can flag nodes PFAIL, but their reports do not count toward the majority and they do not vote in elections. The decision-making body is exactly the set of primaries that own slots.

This is the arithmetic that sets a cluster's minimum size. Three primaries give a majority of two, so one primary can be lost. Adding replicas improves what can be promoted and adds nothing to what can be decided. A cluster with two primaries and six replicas has eight nodes and no working failover, which is a shape nobody designs on purpose and that deployments do drift into by removing a shard.

cluster-node-timeout, the one knob

Defaulting to 15000 milliseconds, cluster-node-timeout sets how long a node must be unreachable before being suspected, and it also bounds several other behaviours, which is why it is the single most consequential cluster setting.

It decides the detection latency, and therefore roughly the outage length for a failed shard. It decides how long a replica waits before it may stand for election. It decides how long a minority-side primary keeps accepting writes before stopping. And it decides how long a node can be unreachable without consequence, which matters for ordinary network variance. One setting moving all four is convenient and means a change to it should be reasoned about from all four directions.

What a sensible value is

Fifteen seconds is conservative and is a long outage for anything in a request path. Five seconds is a common tightening and gives an outage of roughly six or seven seconds. Below a couple of seconds, ordinary network variance and any event-loop stall start producing spurious failovers.

The lower bound in practice is set by the slowest thing that can block a node's event loop, because a node that cannot answer a ping looks dead regardless of why. A node that occasionally spends 800 milliseconds on a large collection operation, or that forks for a snapshot on a huge dataset, needs a timeout comfortably above that, which couples this setting to how large the shards are and how disciplined the commands are.

What makes a node look failed when it is not

A blocked event loop, which SLOWLOG shows. A fork pause on a large dataset. Swapping, which turns every memory access into disk latency. A saturated network link. Or a firewall that closes idle connections on the bus port, which produces a cluster that works until a node goes quiet for a while.

All five produce the same symptom, which is a node flagged PFAIL or FAIL while it is running normally from its own point of view. The diagnosis always starts from the node's own log and SLOWLOG, because the node knows whether it was busy, and ends at the network only if the node believes it was healthy and responsive throughout.

The bus port and the handshake

A node unreachable on the bus port specifically, with the client port open, is reachable by clients and invisible to its peers. It is flagged FAIL and failed over while serving traffic perfectly well, which produces two nodes believing they own the same slots until the configuration propagates.

This is why the bus port appears in every cluster checklist. The symptom is distinctive: a cluster where nodes stay in a handshake state at setup, or where a node is persistently marked failed while clients can reach it. Both are network problems that look like Redis problems, and both are found by checking reachability on the client port plus 10000 between every node pair.

What CLUSTER NODES tells you

Each line carries the node identifier, address with the bus port, flags including myself, master, slave, fail? for PFAIL and fail for FAIL, the primary's identifier for a replica, ping-sent and pong-received timestamps, the configuration epoch, the link state, and the slot ranges.

The ping and pong timestamps are the underused part. A nonzero ping-sent with an old pong-received means a ping is outstanding right now, which is the live view of a link going bad, and comparing those timestamps across nodes localises a problem to a pair rather than to a node. Reading CLUSTER NODES from several nodes and diffing the flags is the first diagnostic step for anything involving a suspected node failure.

How this differs from Sentinel

Sentinel separates the voters from the data nodes, so the voter count and placement are independent of the shard layout, and it exposes a tunable quorum for detection and a fixed majority for action. Cluster makes the primaries the voters and uses one fixed majority for both.

The trade is fewer moving parts against less control. Cluster cannot be misconfigured with a quorum of 1 because there is no quorum to set, and it cannot have its sentinels placed on application servers to see the network the way clients do. Its detection reflects the view of the data nodes, which is usually what you want and occasionally is not, for instance when the data nodes can see each other and the clients cannot see them.

What to take from detection

Three things. The threshold is a majority of slot-owning primaries, fixed, so the primary count is the fault-tolerance arithmetic. cluster-node-timeout governs detection, election delay, the minority write window and tolerance of network variance all at once. And a node that cannot answer a ping is indistinguishable from a dead one, so anything that blocks the event loop is a failover risk.

Key Insight

A node that cannot answer a ping is indistinguishable from a dead node, whatever the reason. A fork pause, a slow collection command, swapping or a saturated link all produce the same PFAIL, which is why the lower bound on cluster-node-timeout is set by the slowest thing that can block a node's event loop rather than by how fast you would like detection to be.