An upgrade is the operation where a cluster's design decisions become visible, because every assumption about queue placement and client reconnection is tested at once.

The version compatibility rules

Two rules that decide whether an in-place upgrade is possible at all.

RabbitMQ supports rolling upgrades between adjacent minor versions, so 3.12 to 3.13 can be done node by node with the cluster serving throughout, while 3.11 to 3.13 cannot and requires either an intermediate hop or a full-cluster stop. Patch versions within a minor are always rolling-upgradable. The jump that catches people is the one that spans two minors, because the intermediate hop is a second full upgrade rather than a formality.

The second rule is the Erlang version. A cluster's nodes must run compatible Erlang releases to form a cluster at all, and a RabbitMQ version has a minimum supported Erlang, so an upgrade frequently implies an Erlang upgrade in the same operation. The Erlang change is the one that can require a full restart rather than a rolling one, depending on the distribution protocol compatibility between the versions involved.

Major-version boundaries, such as 3.13 to 4.0, carry their own rules and usually a published upgrade path with required intermediate versions and prerequisites, including feature flags that must be enabled before the upgrade is attempted. Reading that path rather than assuming adjacency is the difference between an evening and a weekend.

What happens to a node's queues while it is down

Three different answers, which is why the queue type decision made months earlier determines the upgrade's shape.

A classic queue without mirroring lives on one node, and while that node is down the queue is unavailable: publishes to it fail or are unroutable depending on topology, consumers cannot attach, and the messages are on disk waiting for the node to return. The queue comes back intact and the outage is the node's downtime.

A quorum queue with three members continues serving from the remaining two, electing a new leader if the leader was on the restarting node, and the restarting member catches up from the log when it returns. A rolling upgrade is transparent for quorum queues and an outage per node for unmirrored classic queues, which is the single largest practical difference between the two types and the reason the migration in this lesson exists.

A classic mirrored queue is the awkward middle case. It survives the loss of a node if a mirror is elsewhere, but a mirror that comes back has to synchronise from the leader, and depending on the synchronisation policy that is either automatic or requires an explicit action, during which the queue is running with reduced redundancy.

The upgrade methods, and when each one is the only option

Three approaches with different downtime and different risk, and the choice is usually forced rather than free.

A rolling upgrade is the default and the cheapest: one node at a time, the cluster serving throughout, and the only unavailability is per-queue and per-node. It requires adjacent versions and compatible Erlang, and it requires that the cluster can survive the loss of one node, which a two-node cluster cannot do for quorum queues at all since two members have no majority without both.

A full-cluster stop is the fallback when the version gap or the Erlang change forbids a rolling upgrade. Every node stops, every node is upgraded, and the cluster restarts, which means the downtime is the full upgrade duration rather than a per-node restart. The compensating property is that there is never a mixed-version cluster, so the entire class of mixed-version problems does not apply, which is why a full stop is sometimes the right choice even when a rolling upgrade is technically possible.

Blue-green is the third: stand up a new cluster on the new version, move producers and consumers across, and decommission the old one. It costs double the hardware for the duration and it is the only approach that offers a real rollback, since the old cluster is still running and still has its data. For a major-version jump or a cluster whose upgrade has no tested path, that rollback is worth the hardware.

Backups, and what a RabbitMQ backup actually covers

One distinction that decides whether a rollback plan exists.

RabbitMQ's definitions, meaning the vhosts, exchanges, queues, bindings, users, permissions, policies and parameters, can be exported and imported as JSON with rabbitmqctl export_definitions and import_definitions, and that export is small, fast and should be taken before every upgrade. It is the thing that lets a rebuilt cluster come back with the right topology rather than an empty broker.

The messages are a different matter. The message store is a set of on-disk files under the node's data directory, and it is not portable between versions in general, so a definitions export is a topology backup and not a data backup, and the only reliable way to preserve messages across a failed upgrade is to have drained the queues first or to have a second cluster holding them.

The practical consequence is that the pre-upgrade checklist has a step most people skip: drain what can be drained. A queue that is empty at the start of the upgrade cannot lose messages during it, which turns the hardest part of the risk assessment into a non-question for every queue whose consumers can keep up.

What to verify before starting

A short list, each item of which has blocked an upgrade in practice.

The target version's release notes and upgrade path, specifically whether the jump is adjacent and whether any feature flag is listed as required. The Erlang version requirement, checked against what is installed and what the operating system's packages can supply, since an Erlang upgrade is sometimes the larger piece of work. The current feature flag state, because any flag not enabled is a potential blocker rather than a note.

Then the cluster's own readiness: every node running and clustered, no quorum queue short of its declared members, no classic mirror unsynchronised, and no resource alarm in effect. An upgrade started on a cluster that is already degraded has no safe stopping point, because the first node stop takes it below the threshold that the degradation already consumed.

The last item is the client inventory: which applications connect, whether each reconnects automatically, and who to notify. That list is tedious to assemble and it is the one that determines whether the upgrade produces an incident report, which makes it worth more than its position at the end of a checklist suggests.

The order of operations

A sequence whose steps each exist because of a specific failure.

Upgrade the node that is least critical first, which in a cluster with an uneven queue distribution is a real choice rather than an arbitrary one. Before stopping each node, confirm it is not quorum-critical, which is the check from the previous lesson and the step most often skipped. Stop the node with rabbitmqctl stop_app or a clean service stop rather than a kill, so that queues flush and the node records a clean shutdown.

Install the new version, start the node, and then wait for it to fully rejoin before touching the next one. Fully rejoining is more than the process being up: a node is not ready for the next step until its quorum queue members have caught up and its classic mirrors have synchronised, and a procedure that waits only for the port to open starts the next step while the cluster is still at reduced redundancy.

The last node has one special property worth knowing. The node stopped last should be the node started first after a full-cluster stop, because it holds the most recent schema state, and starting a different node first means it waits for the one with the authoritative state. That rule does not apply to a rolling upgrade and applies absolutely to a full stop, which is why the two procedures cannot be treated as variations of each other.

What clients experience

The part that produces the incident report, usually independent of whether the upgrade itself went well.

Every client connected to the restarting node has its connection closed. Well-behaved clients reconnect with backoff, rediscover their queues and resume, and the effect is a brief error rate. Clients that do not implement reconnection stay down until restarted, which is a client-library and configuration question rather than a broker one, and the usual discovery during the first upgrade of a new system.

Two subtler effects matter more. Consumers that reconnect do so all at once, which produces a thundering herd of channel and queue declarations against a node that has just started, and on a large cluster that is enough load to slow the start. And a reconnecting consumer loses its unacknowledged messages, which are redelivered, so an upgrade produces a burst of redeliveries and any non-idempotent consumer will show it.

The mitigation for the first is staggered reconnection with jitter in the client configuration. The mitigation for the second is idempotent consumers, which is the correct design regardless, and the upgrade is simply the event that reveals whether they are.