Replication gives you a replica holding the same data. It does not notice that the primary has died, choose which replica should take over, perform the promotion, or tell the clients. Sentinel is the piece that does those four things.

The gap replication leaves

A primary with two replicas survives the loss of the primary only in the sense that the data survives. Somebody still has to decide that the primary is actually gone rather than briefly unreachable, pick a replica, issue REPLICAOF NO ONE, re-point the other replica, and get every client to connect somewhere new.

Done by a person, that is several minutes at best and relies on someone being awake. Done by a script, it is a script that has to decide alone whether a primary is down, which is exactly the decision that cannot safely be made alone: a script running on a host with a broken network concludes the primary is dead when in fact the script is the isolated party. The need for agreement is the whole reason Sentinel exists in the shape it does.

What a sentinel actually is

A sentinel is the same redis-server binary started with --sentinel, or redis-sentinel which is the same thing. It holds no dataset, accepts a small dedicated command set, and listens on port 26379 by default.

Because it is an ordinary Redis process speaking the ordinary protocol, you can connect to it with redis-cli -p 26379 and ask it anything. That is worth knowing because it makes the entire state of a Sentinel deployment directly inspectable: SENTINEL masters, SENTINEL replicas <name> and SENTINEL sentinels <name> tell you what each sentinel believes, and comparing those beliefs across sentinels is how almost every Sentinel problem is diagnosed.

The four jobs

Monitoring, which is checking that the primary and its replicas are reachable. Notification, which is reporting state changes through pub/sub and optional scripts. Automatic failover, which is promoting a replica and reconfiguring the rest. And acting as a configuration provider, which is answering the question "where is the primary right now".

The fourth is the one people underestimate. Even with a perfect failover, a client holding a hardcoded primary address learns nothing and keeps failing. Sentinel's value is only realised if clients ask it where to connect, which means a sentinel-aware client library rather than a connection string, and that is a change to the application rather than to the infrastructure.

Why there are at least three

A single sentinel is a single point of failure and, worse, cannot distinguish its own isolation from the primary's death. Two sentinels can disagree with no way to resolve it. Three is the smallest number that lets a majority exist when one is lost.

The majority requirement is not about the failure detection, which is covered by the configurable quorum, but about authorising a failover. A failover needs a majority of the known sentinels to agree on a leader, and with two sentinels a majority is two, so losing either one makes failover impossible. Three sentinels tolerate one loss. Five tolerate two. The pattern is the familiar one and the reason is the familiar one.

Where the sentinels go

On three separate hosts, ideally in three separate availability zones, and specifically not all on the same host as the Redis instances they monitor. A deployment with three sentinels on the primary's host loses all three when the host dies, which is the exact moment they were needed.

Common practical placements are a sentinel alongside each Redis instance when there are three or more instances, or sentinels on the application servers, which has the pleasing property that a sentinel sees the network the way a client does. The arrangement to avoid is one that correlates sentinel failures with Redis failures, and the question to ask of any proposed layout is which single failure takes out both a sentinel majority and the primary.

The configuration is tiny

A sentinel needs to be told which primary to watch, and nothing about the replicas or the other sentinels.

1port 26379
2sentinel monitor mymaster 10.0.1.10 6379 2
3sentinel down-after-milliseconds mymaster 5000
4sentinel failover-timeout mymaster 60000
5sentinel parallel-syncs mymaster 1

The 2 on the monitor line is the quorum, the number of sentinels that must agree the primary is unreachable. The name mymaster is the handle clients and commands use. Everything else in the deployment, the replicas and the other sentinels, is discovered rather than configured, which is both convenient and occasionally confusing when a stale discovery persists.

How replicas are discovered

A sentinel connects to the primary and reads INFO replication, which lists the connected replicas with their addresses. It then monitors those too, and repeats the check every ten seconds so that a newly attached replica is picked up automatically.

This has a consequence worth understanding. The addresses Sentinel learns are the ones the primary reports, which are what each replica announced during its handshake. A replica behind NAT or in a container that announces an unreachable address is discovered at that unreachable address, and Sentinel then considers it permanently down while ordinary replication works perfectly. replica-announce-ip and replica-announce-port exist for exactly this case, and this is the commonest Sentinel problem in containerised deployments.

How sentinels discover each other

Every sentinel publishes a message to the __sentinel__:hello channel on the monitored primary and every monitored replica, every two seconds, carrying its own address, its run identifier, its current epoch and its view of the primary. Every sentinel also subscribes to that channel.

So the Redis instances act as a meeting place, and sentinels learn about each other without being configured with each other's addresses. Adding a sentinel is therefore a matter of starting it with the primary's address, and it joins. Removing one is less automatic, because the others remember it, which is what SENTINEL reset is for and is the reason a decommissioned sentinel can keep a majority out of reach.

The configuration file is rewritten

Sentinel writes its discovered state back into its own configuration file: the current primary address, the known replicas, the known sentinels and the current epoch. The file is not the static input it looks like.

Two consequences follow. The file must be writable, and a sentinel started with a read-only configuration file refuses to run. And a configuration file copied from one sentinel to another carries a run identifier and discovered state that belong to the first, so templating sentinel configuration with a full copy of a running sentinel's file produces subtle duplicate-identity problems. Deploy the minimal five lines and let discovery fill in the rest.

What Sentinel does not do

It does not shard, so the whole dataset still lives on one primary. It does not proxy, so clients still connect directly to Redis and must learn the address. It does not make replication synchronous, so a failover still loses whatever the promoted replica had not received.

That last point is the one to be clear about, because "high availability" is sometimes heard as "no data loss". Sentinel reduces the duration of an outage from minutes to seconds. It does nothing at all about the acknowledged writes that were in flight when the primary died, which remains a property of asynchronous replication and is bounded only by min-replicas-to-write and by WAIT.

What the rest of this lesson covers

The second section is how sentinels decide a primary is down, which is where the quorum and the majority are two different numbers that people routinely conflate. The third is the failover itself, including how a replica is chosen. The fourth is what clients see and what the window costs them. The fifth is operating a Sentinel deployment, which is mostly about placement, testing and a short list of misconfigurations that quietly disable the whole thing.

Those four sections are in the order a failover happens, which is also the order the diagnosis goes when a failover does not happen or happens badly.