Persistence and Memory Behaviour

Topics Covered

What Durable Actually Means

Two flags, four combinations

Quorum queues and streams remove the question

Persistent does not mean written yet

The unacknowledged window on the consuming side

Lazy declaration, and the equivalence check

Policies cannot rescue a declaration

What a restart loses, precisely

Where this leaves the single-node broker

Durability in RabbitMQ is two independent settings that are routinely discussed as one, and almost every "we lost messages but everything was durable" incident traces to the gap between them.

Two flags, four combinations

And only one of the four survives a restart.

A queue is durable or not, which is a property of the queue declaration and decides whether the queue itself exists after a broker restart. A message is persistent or not, set per publish through delivery_mode with 2 meaning persistent and 1 meaning transient, and it decides whether the message is written to disk.

A transient message in a durable queue is gone after a restart; the queue comes back empty. A persistent message in a non-durable queue is also gone, because the queue does not come back and its contents go with it. Only persistent messages in a durable queue survive, and that combination has to be chosen on both sides by two pieces of code that are often written by different people.

The fourth combination, transient in non-durable, is the fast path and is honest about what it is. The trap is not that combination; it is the two mixed ones, because each looks durable from one side.

Quorum queues and streams remove the question

Which is a feature, not a limitation.

A quorum queue is always durable and treats every message as persistent, so delivery_mode is effectively ignored. A stream is the same. Neither type offers a transient mode, which means the two mixed combinations above cannot be constructed on them at all.

That is the strongest argument for the newer types in a system where durability matters: the configuration cannot be wrong, because there is no configuration. A classic queue requires that every publisher sets delivery_mode correctly forever, including publishers added later by teams that did not read the original design, and a single publisher defaulting to transient produces intermittent loss that reproduces only across a restart.

The cost is that the floor is a replicated disk write and does not move. A quorum queue cannot be made cheaper by accepting less durability, because that trade is not exposed. For a work queue whose contents matter that is the right default; for high-volume data that is genuinely disposable, a classic queue with transient messages is still the cheaper structure and the choice is deliberate rather than careless.

Persistent does not mean written yet

The distinction that creates the loss window.

Setting delivery_mode to 2 makes a message eligible to be written to disk. It does not make the write synchronous with the publish, and basic.publish has no response, so a publisher that sets persistence and sends is told nothing about whether the message reached disk, reached the broker, or reached anything at all.

The broker batches writes, so there is a window between accepting a message and having it durably stored. A broker that is killed inside that window loses persistent messages from a durable queue, and the publisher has no way to know which ones.

Publisher confirms close that window and nothing else does. A confirm for a persistent message in a durable classic queue means the message is on disk; for a quorum queue it means it is in the logs of a majority of members; for a stream it means the chunk is written and replicated to a quorum. In every case the confirm is the event that moves responsibility from the publisher to the broker, and without it the publisher is responsible for a message it cannot see.

Two independent flags, four combinations, one survivor. The two mixed combinations each look durable from one side, which is why they produce loss that reproduces only across a restart.

The unacknowledged window on the consuming side

A second gap, symmetrical with the first.

A message delivered to a consumer but not yet acknowledged is still owned by the broker and will be redelivered if the consumer's channel closes. That is the mechanism working. The loss case is autoAck, where the broker removes the message at the moment of delivery and the consumer's crash takes it with it.

autoAck is not a tuning option; it is a delivery-semantics choice. It converts the pipeline to at-most-once regardless of how durable everything else is, and a system with persistent messages in quorum queues consumed with autoAck has paid full price for durability and then discarded the guarantee at the last hop.

The symmetry is worth holding onto: the broker's guarantee covers the span between a confirm and an acknowledgement, and both ends are the application's responsibility. Everything in this lesson about disks and memory concerns what happens inside that span, and none of it helps if either end is open.

Lazy declaration, and the equivalence check

How a durability setting gets fixed by whoever declared first.

A queue's durability is set at declaration and cannot be changed afterwards. Redeclaring an existing queue with different properties does not reconfigure it; the broker performs an equivalence check and returns a channel-level error, PRECONDITION_FAILED, closing the channel.

That makes the first declaration authoritative, and in a system where several services declare the same queue at startup it is a race. A service that declares the queue as non-durable can win, and every other service then fails its own declaration and, depending on how its client handles a channel error, either crashes loudly or reconnects and retries forever.

The loud failure is the good outcome, because it is visible. The bad outcome is a client that catches the error, continues on a fresh channel, and publishes successfully into a queue whose durability is not what its author intended. Declaring topology from application code makes durability a property of deployment order, which is the argument for a definitions file loaded by the broker instead.

Policies cannot rescue a declaration

A distinction that catches people who expect otherwise.

Policies change a great deal about a queue after the fact: retention, length limits, overflow behaviour, dead-letter routing, delivery limits. They cannot change durability, queue type, or exclusivity, because those are structural and the queue's storage was laid out according to them.

So the repair for a wrongly declared queue is to delete and redeclare it, which means dealing with its contents, or to declare a correctly configured queue beside it and migrate consumers and then publishers. There is no configuration change that converts a non-durable queue into a durable one.

This is worth knowing before an incident rather than during one, because the instinct under pressure is to look for the setting that fixes it. The useful mental split is structural at declaration, behavioural by policy, and durability, type and exclusivity are all in the first group.

What a restart loses, precisely

Three categories, often conflated.

A clean shutdown through rabbitmqctl stop or a service stop flushes what it has and loses nothing that was confirmed. The broker writes out its state deliberately, which is why a planned restart and a crash are different events with different consequences.

A process crash, meaning the Erlang VM dies while the operating system survives, loses nothing that was written to the file system, because the page cache is the OS's and outlives the process. Unconfirmed in-flight messages are lost; confirmed ones are not.

A machine loss, meaning power failure or a hard reset, loses whatever was in the page cache but not yet synced to the device. This is the only one of the three where fsync behaviour matters, and it is the case the next section is about. For a single-node broker it is the case that can lose confirmed messages; for a quorum queue or stream with members on separate machines, it is survivable because the majority holds the data.

Where this leaves the single-node broker

A clear statement, because it is often left implicit.

A single-node RabbitMQ with durable queues and persistent messages survives process crashes and clean restarts fully. It survives a machine loss only as far as its last sync, and it does not survive the loss of the machine itself at all, because there is nowhere else the data is.

Durability protects against restarts; replication protects against losing the node. A classic queue, however durable, is a single-node structure, which is the whole reason quorum queues and streams exist. An architecture document that lists "durable queues" as its availability story has conflated the two.

That is also why the mirrored-queue era mattered and why its removal was not a regression. Mirroring tried to add replication to a structure whose replication was not consensus-based, which meant a partition could lose acknowledged messages. Quorum queues and streams replace it with replication that cannot, and the price is that both are always durable, which brings the argument back to the first subsection: the configuration cannot be wrong because there is none.

The broker's guarantee covers the span between a confirm and an acknowledgement. Both ends belong to the application, and autoAck or missing confirms discards the guarantee regardless of how durable the storage is.
Common Pitfall

Durable queue and persistent message are two independent settings and only the combination of both survives a restart. A transient message in a durable queue is gone and so is a persistent message in a non-durable queue, and each of those looks durable from one side, which is why the loss reproduces only across a restart. Quorum queues and streams remove the question entirely by being always durable, which is the strongest argument for them in a system where contents matter: the configuration cannot be wrong because there is none.