Quorum Queues and the Raft Log

Topics Covered

Raft, as RabbitMQ Implements It

One Raft group per queue

Why the replica count should be odd

The leader, and what it does alone

Terms, and the number that explains an incident

Members are processes, not copies of a file

Always durable, and what that removes

A quorum queue is not a classic queue with replication bolted on. It is a replicated log with a consensus protocol underneath it, and almost every difference in its behaviour, including the ones that look like limitations, follows from that.

One Raft group per queue

The library is ra, and the unit is the queue.

Declaring a queue with x-queue-type set to quorum creates a Raft cluster whose members are processes on several nodes, one of which is the leader and the rest followers. Each member holds a copy of the queue's log, and the log is the authoritative record: every publish, every delivery, every acknowledgement and every requeue is an entry appended to it.

The implementation is Erlang's ra library, a Raft implementation maintained alongside RabbitMQ. The important structural point is that the Raft group is per queue, not per node or per cluster, so a broker with 500 quorum queues is running 500 independent Raft groups, each with its own leader, its own election timeouts and its own log. That is a lot of groups, and it is the reason quorum queues have a per-queue overhead that classic queues do not.

The group's size is fixed at declaration. x-quorum-initial-group-size, or the quorum_cluster_size node setting, defaults to 3, or to the cluster size if the cluster is smaller. Members are chosen when the queue is declared and do not change when nodes are added to or removed from the cluster, which is one of the most consequential operational facts in this lesson.

Why the replica count should be odd

Because a majority of an even number is the same as a majority of the next odd one.

Raft requires a majority of the group to agree before an entry is committed. With 3 members a majority is 2, so the group tolerates 1 failure. With 5 members a majority is 3, so it tolerates 2. With 4 members a majority is 3, so it also tolerates only 1, exactly like 3 members, while paying for a fourth copy of every message on every write.

So an even group size is strictly worse than the odd size below it: the same fault tolerance, more storage, more network traffic and more latency per write. This is not a subtlety, it is arithmetic, and the only reason it ever happens is a cluster with an even node count and a policy that set the group size to the cluster size.

Three is the right default for almost everything. Five is for queues whose unavailability is genuinely worse than the additional write cost, and it doubles the fault tolerance at a cost of roughly 67% more replication traffic per message. Beyond five the write latency grows and the fault tolerance gained is rarely worth it, because losing three nodes of five is usually a cluster-wide event rather than an independent-failure one.

One Raft group per queue. An entry is committed when a majority of members have it, which is 2 of 3 or 3 of 5, so a group of 4 tolerates exactly the same single failure as a group of 3 while paying for a fourth copy.

The leader, and what it does alone

Everything client-facing goes through one member.

All publishes and all deliveries for a quorum queue are handled by its leader. A client connected to a node holding a follower has its operations forwarded to the leader, which means a cross-node hop, exactly as with a classic queue on another node. Followers are not read replicas; they exist to hold the log and to be electable.

The leader therefore carries the per-queue work, and the placement question returns in a new form. queue_leader_locator controls where a new queue's leader is placed, with client-local putting it on the declaring client's node and balanced spreading leaders across the cluster. For an estate with many quorum queues, balanced is the setting that matters, because leaders concentrated on one node concentrate both the client-facing work and the replication fan-out.

When a leader fails, the followers hold an election and one of them becomes the new leader, typically within a few seconds. Clients see their consumers cancelled and reattach, publishers see confirms resume, and messages that were in flight are redelivered from the log. That recovery is the whole point of the type: no operator action, no lost acknowledged messages, and a bounded window of unavailability rather than an outage lasting until a node is repaired.

Terms, and the number that explains an incident

Every election increments a counter that never goes backwards.

Raft divides time into terms, each with at most one leader. A term begins when a member decides the leader is gone and stands for election, incrementing the term number and asking the others to vote. A member votes for at most one candidate per term, which is what prevents two leaders in the same term, and a candidate that receives a majority of votes becomes leader for that term.

The term number is the single most useful diagnostic number in a quorum queue incident, because it is a count of how many times the group has decided its leader was gone. A queue with a term of 3 after six months is a healthy queue. A queue whose term is in the hundreds has been electing repeatedly, and the cause is almost always environmental rather than queue-specific: a node that is up but slow enough to miss heartbeats, a network with intermittent loss, or a node under memory pressure whose Erlang scheduler is not running the Raft timers promptly.

This failure mode is worth naming because it is both common and self-worsening. Each election costs a round of messages, cancels consumers, fails in-flight publishes and forces clients to reattach, and the reattachment load lands on a node that was already struggling. An estate with hundreds of quorum queues on a marginal node does not see one election; it sees hundreds, concurrently, repeatedly. The fix is never in the queue's configuration.

Members are processes, not copies of a file

Which is why a member can be present and useless.

It is tempting to think of a group's members as three copies of the same data, with health being a question of whether all three exist. The members are Erlang processes, each running the Raft state machine, and each has its own position in the log: a commit index, the highest entry it knows is committed, and a last-applied index, the highest entry it has actually applied to the queue state.

A member can therefore be in several distinct conditions. Absent, because its node is down. Present and current, with its indexes tracking the leader's. Present and lagging, with indexes persistently behind, which is a member that answers heartbeats and votes but whose copy of the data is incomplete. Or recovering, catching up from the leader's log or from a snapshot after being away.

The lagging case is the one that defeats naive monitoring. The queue reports three members, the cluster reports three healthy nodes, and the group's effective redundancy is two, because the third member's log would not satisfy the election safety requirement for the entries it is missing. Checking member count is not checking replication; checking index convergence is.

The term number counts how many times the group decided its leader was gone. A term in the hundreds is a marginal node or a lossy network, not a queue misconfiguration, and the reattachment load from hundreds of simultaneous elections lands on the node that was already struggling.

Always durable, and what that removes

There is no transient quorum queue.

A quorum queue is durable by definition, because its contents are a Raft log and a Raft log is written to disk. The durable flag on queue.declare must be true, and delivery_mode on a published message is effectively ignored: a transient message published to a quorum queue is written to the log exactly like a persistent one.

That removes a whole category of tuning. There is no decision about whether a message is persistent, no transient store, no paging policy, and no queue-mode. It also removes a performance option: a classic queue carrying genuinely worthless data can be transient, shallow and extremely fast, and a quorum queue carrying the same data cannot. Choosing a quorum queue for data that does not matter is paying a replication cost for nothing.

The corollary is that a quorum queue's floor cost per message is a disk write replicated to a majority, and that floor does not move. Every performance conversation about quorum queues happens above that floor, which is why their raw throughput is generally below a classic queue's for the same shape of work and why the choice between them is a correctness decision rather than a performance one.

Key Insight

A quorum queue is one Raft group per queue, so a broker with 500 quorum queues runs 500 independent Raft groups with their own leaders, timeouts and logs. The group size is fixed at declaration and does not change when cluster membership changes. An even group size is strictly worse than the odd size below it: 4 members tolerate the same single failure as 3 while paying for a fourth copy of every write.