A RabbitMQ cluster is a much weaker form of cooperation than the word suggests, and most partition surprises come from assuming it shares more than it does.

The metadata store, and what lives in it

One replicated database holding definitions, not messages.

Every node in a cluster holds a full copy of the metadata: virtual hosts, exchanges, queues, bindings, users, permissions, policies and parameters. Historically this was Mnesia, Erlang's built-in distributed database; current versions use Khepri, a Raft-based store that replaced Mnesia specifically because Mnesia's partition behaviour was the source of most of the problems in this lesson.

What the metadata does not hold is messages. A queue's definition is replicated to every node, and its contents are not: a classic queue's messages live on exactly one node, the one where the queue was declared, and a quorum queue's live on the members of its Raft group. Knowing a queue exists and being able to reach its messages are two different things, and a cluster guarantees the first everywhere while the second depends on which node you reach.

That split is the single most useful fact about a cluster. A client connected to node B can declare, bind and publish to a queue whose contents are on node A, because the metadata is local and the publish is forwarded. If node A becomes unreachable, the metadata still says the queue exists and the publish now fails, which is why a partition produces errors that reference queues the client can plainly see.

Which operations need agreement

A short list, and it is the list that fails during a partition.

Declaring or deleting an exchange, a queue, a binding, a user or a policy changes the metadata, and under Khepri that is a Raft write requiring a majority of nodes. Those operations fail on a node that cannot reach a majority, with an error rather than a timeout in recent versions, which is a considerable improvement over Mnesia's behaviour of succeeding locally and diverging.

Publishing and consuming need no agreement. They need the queue's own node, or for a quorum queue a majority of that queue's members, which is a different and usually smaller set than the cluster. A three-node cluster with a partition isolating one node can continue serving every quorum queue whose majority sits in the larger side, with no metadata operations possible on the isolated node and full message flow on the other.

The practical consequence is a failure signature worth recognising: an application that can publish and consume but cannot declare is talking to a minority node. Clients that declare their topology on every connection, which is the common defensive pattern, convert that into a total outage for themselves, because the declare fails before any publishing is attempted.

What the inter-node link actually carries

Four kinds of traffic on one connection, and they compete.

The Erlang distribution link between two nodes carries everything: metadata replication, forwarded publishes for queues whose data is elsewhere, quorum queue and stream replication traffic, management plugin metric collection, and the periodic ticks that decide whether the peer is alive. It is one TCP connection per node pair by default, multiplexing all of it.

That multiplexing is why a heavy replication load can cause a detected partition. A quorum queue replicating large messages saturates the same link the ticks travel on, and a tick delayed past its deadline is indistinguishable from a tick that never arrived. The distribution link is also where head-of-line blocking appears: a large message being replicated occupies the link while a small metadata write waits behind it.

The practical consequences are two. First, inter-node traffic scales with how much work is done on the wrong node: a client publishing to a queue whose data is on another node pays a forwarded hop per message, so a load balancer distributing connections uniformly over a cluster whose queues are not uniformly distributed generates forwarding traffic proportional to the mismatch. Second, the link's capacity is a real constraint that no RabbitMQ metric reports directly, which is why a cluster that behaves badly under load with no visible broker-side cause is worth investigating at the network layer before anywhere else.

Virtual hosts, and the scope of everything

One namespace boundary, applied cluster-wide.

Every exchange, queue, binding and policy lives inside a virtual host, and a virtual host exists on every node in the cluster because it is metadata. A connection selects its virtual host at connection.open and cannot cross between them afterwards, so routing a message from one virtual host to another requires a shovel or federation link even within a single broker.

That makes a virtual host the unit of logical isolation and not of failure isolation. Two virtual hosts on the same cluster share the nodes, so they share the memory alarm, the disk alarm, the distribution links and the partition behaviour. A virtual host separates permissions and names, not resources, which is why using virtual hosts to isolate a noisy tenant does not work and separate clusters do.

Where it does matter for this lesson is that default_queue_type is a virtual-host-level setting, so the decision to make unspecified queues quorum queues is made per virtual host rather than per cluster. A cluster with twelve virtual hosts has twelve places to set it, and the one that was missed is where the exposed classic queues are.

Node types, and the one that no longer matters

A distinction worth knowing because the documentation and the tooling still carry it.

Mnesia-era clusters distinguished disc nodes, which persisted metadata to disk, from RAM nodes, which held it only in memory and depended on a disc node for durability. The guidance was to run at least two disc nodes so that losing one did not lose the metadata, and RAM nodes were offered as a throughput optimisation for workloads doing heavy queue churn.

Under Khepri the distinction is gone: every node is a Raft member with its own on-disk log. rabbitmqctl cluster_status still reports node types in some versions and the --ram flag still exists in others, which produces the impression of a choice that no longer has consequences. A current cluster has one node type, and advice about disc-node counts applies to deployments old enough that the metadata store is still Mnesia.

What survives from that era is the shape of the advice rather than its content. The reason two disc nodes were recommended is the reason an odd number of nodes is recommended now: the metadata store needs a majority to accept a write, so the cluster's tolerance for node loss is a property of its size and nothing else.

Forming a cluster, and how nodes find each other

Three mechanisms, and the one that fails quietly.

A node joins by being told to, with rabbitmqctl join_cluster naming an existing member, which stops the node, discards its local metadata and adopts the cluster's. The discarding is total and is the usual source of a lost-queues incident during a cluster rebuild: joining is not merging, and a node with queues of its own loses them.

Peer discovery automates the same thing for orchestrated deployments, through backends for DNS, Kubernetes, Consul, etcd and AWS. The mechanism is the same join under the covers, and the backend only supplies the list of peers to try. Discovery is where a cluster can silently form as several single-node clusters instead of one, because a node that fails to reach any peer within its retry window starts standalone, and a standalone node is a working broker that serves traffic and reports healthy.

All of this depends on the Erlang distribution protocol, which requires a shared secret. Every node must hold the same Erlang cookie, and a mismatch produces a connection failure that looks like a network problem and is reported in the log as an authentication failure between nodes. Checking cookie equality is the first step in diagnosing a cluster that will not form, before any network investigation, because the symptom is identical and the cause is far more common.

1# what the cluster thinks it is, run on every node and compared
2rabbitmqctl cluster_status
3
4# the node's own view of the Erlang distribution layer
5rabbitmq-diagnostics ping
6rabbitmq-diagnostics check_running
7rabbitmq-diagnostics node_health_check
8
9# where a queue's messages actually are
10rabbitmqctl list_queues name type leader members --formatter=pretty_table

The last of those is the command worth running before any partition discussion, because it answers the only question that matters during one: for each queue, which nodes can serve it. A cluster whose queues are all classic and all on one node is not a highly available deployment regardless of how many nodes it has.