0%
RabbitMQ Deep Dive: AMQP, Queues, and Delivery Guarantees
The Broker Model
Queues and Storage
Flow and Scale
Patterns
Publisher Confirms and Transactions
basic.publish has no response, which is the single most consequential fact in AMQP. Confirms are the extension that gives it one, and the shape of that extension decides what a publisher has to do.
Why publish has no reply
A deliberate asymmetry, not an oversight.
AMQP's method classes mostly come in request and response pairs, and basic.publish is one of the exceptions. The reason is throughput: a synchronous reply per publish would make the protocol's rate a function of round-trip latency, and a broker on a 1 ms network would be capped near 1,000 messages per second per channel no matter how fast it was.
So the wire protocol gives the publisher no indication of anything. A publish that is routed to three queues, a publish that matches no binding and is discarded, and a publish on a connection that is about to be closed all look identical from the publisher's side, which is a message sent into a socket with no feedback.
Confirms restore feedback without restoring synchrony, which is why they are an extension rather than a change to basic.publish. The publisher keeps writing and receives acknowledgements asynchronously, which preserves pipelining.
Turning them on, per channel
One method, and it is irreversible for that channel.
confirm.select puts a channel into confirm mode and the broker replies confirm.select-ok. The mode applies to the channel, not the connection, and it cannot be turned off; a channel in confirm mode stays in confirm mode until it closes.
A channel in confirm mode also cannot be in transaction mode, and vice versa. Attempting both is a channel-level error, which is the protocol stating that these are two alternative answers to the same problem rather than complementary features.
Because the mode is per channel, a publisher can keep a confirm-mode channel for messages that matter and a plain channel for messages that do not. That is a reasonable design and it is rare in practice, because the cost of confirms is paid in latency rather than in the broker, and most applications find one mode simpler to reason about than two.
Delivery tags, which are per channel and monotonic
The numbering that makes asynchronous confirms possible.
Once a channel is in confirm mode, every published message is assigned a delivery tag, a 64-bit integer starting at 1 and incrementing by one per publish on that channel. The broker's confirms reference these tags, and the publisher has to maintain the mapping from tag to message itself.
The tags are per channel and the sequence restarts at 1 on a new channel, so a publisher that reconnects cannot carry its tag map across. Everything unconfirmed at the moment a channel closes is in an unknown state and has to be handled, which is the subject of the final section.
The broker also never skips a tag, so a publisher can use the sequence as a completeness check. A tag that is neither acked nor nacked and never will be is the failure case, and it exists only when the channel closes, which is what makes close handling the whole of a publisher's error path.
basic.ack, and the multiple bit
The optimisation that catches naive implementations.
A confirm arrives as basic.ack carrying a delivery tag and a multiple flag. With the flag clear, the method confirms exactly the one tag. With it set, it confirms every unconfirmed tag up to and including that one, which lets the broker confirm a thousand messages in a single frame.
A publisher that reads the tag and ignores the flag will leave 999 of those thousand in its unconfirmed map forever, and the symptom is a map that grows without bound while the broker believes everything is confirmed. It is one of the most common bugs in hand-written confirm handling, and it does not appear under light load because the broker has no batches to coalesce.
The flag is also why confirms arrive in an order the publisher cannot predict. A message routed to a slow quorum queue can be confirmed after a message published later to a fast classic queue, so the correct implementation removes a range from a sorted structure rather than assuming the next tag is the next confirm.
The tag the publisher assigns itself
A detail that causes real bugs in multi-threaded publishers.
The broker does not tell the publisher a message's delivery tag. The publisher derives it by counting its own publishes on that channel, starting at 1, and the broker's counting is expected to agree. Nothing on the wire reconciles the two.
That makes the counter and the publish a single operation that must not be interleaved. Two threads publishing on one channel without a lock will assign tags in one order and have their frames arrive in another, so every confirm afterwards is attributed to the wrong message, and the symptom is a publisher that reports success for messages that failed and retries messages that succeeded.
The standard advice that a channel must not be shared across threads exists largely because of this. It is usually stated as a client library constraint, which understates it: even a thread-safe library cannot reconcile a tag sequence it did not assign, so the constraint is in the protocol rather than in the implementation.
Confirms and the channel's other traffic
Why a busy channel confirms more slowly.
Confirms share the channel with everything else on it, including consumer deliveries and acknowledgements if the channel is also consuming. A channel doing both has its confirms queued behind delivery frames, which adds latency that has nothing to do with storage.
This is the practical argument for separate channels for publishing and consuming, and it is stronger than the usual style preference. A consumer with a prefetch of 1,000 can have a thousand delivery frames ahead of a confirm, and the publisher's measured confirm latency then reflects the consumer's throughput.
The same reasoning applies to connections under heavy load, since a connection has one socket and one writer process. A publisher that cares about confirm latency gets its own connection, which costs a few megabytes of broker memory and removes a whole class of unexplained latency.
basic.nack, and what it actually means
Rarer than expected, and more serious.
basic.nack is the negative confirm, carrying the same tag and multiple semantics plus a requeue field that is meaningless in this direction. It means the broker could not take responsibility for the message, and in practice it is emitted when an internal error prevented the message from being stored.
It is not the response to an unroutable message, which is confirmed positively, and it is not the response to a full queue, which is handled by overflow behaviour. Nacks are genuinely uncommon, and a publisher seeing them regularly has a broker problem rather than an application one.
Because they are rare, the nack path is the least tested part of most publishers, and the one most likely to be a TODO. The correct handling is to treat a nack as a failed publish and retry it, which is the same action as a timeout, so the simplest correct implementation has one path for both rather than a special case that was never exercised.
Confirms and the storage layer
What the confirm is actually waiting for, per type.
For a persistent message in a durable classic queue, the confirm follows the relevant flush, so it is a statement that the message is on disk. For a quorum queue it follows commit, meaning a majority of members hold the entry. For a stream it follows the chunk being written and acknowledged by a quorum of replicas.
For a transient message, or a message in a non-durable queue, the confirm follows the message being accepted into the queue and nothing more, because there is no disk write to wait for. That makes confirms fast and makes their guarantee much weaker, which is the correct behaviour and a frequent source of misplaced confidence.
Routing to several queues produces one confirm after all of them have taken responsibility, so the latency is the slowest destination's. A message routed to one classic queue and one quorum queue is confirmed at the quorum queue's speed, which is the mechanism by which adding a durable destination to an existing exchange slows down publishers that had nothing to do with the change.
The unconfirmed window as a resource
Why the publisher's map is a capacity question.
Every unconfirmed message is held by the publisher, because it may have to be republished. The number outstanding is bounded by how fast confirms return, so a slow broker causes a publisher-side memory growth that has nothing to do with the broker's memory.
That makes the unconfirmed window a quantity to bound explicitly. A publisher with an unbounded window and a broker in a disk alarm will accumulate every message it has sent since the alarm began, and the failure that gets reported is a publisher running out of memory rather than a broker refusing writes.
A bounded window converts the broker's backpressure into the publisher's backpressure, which is what it is for. The publisher stops accepting new work when the window is full, the pressure reaches whatever is feeding it, and the system slows down instead of one component failing.
basic.ack carries a multiple flag, and with it set the method confirms every unconfirmed tag up to and including the one named. A handler that reads the tag and ignores the flag leaves everything below it unconfirmed forever, so the map grows without bound while the broker believes the work is done. The bug does not show under light load, because the broker has no batches to coalesce, and it appears the first time the publisher is busy.