Dead-Letter Exchanges and Retry Topologies

Topics Covered

How Dead-Lettering Actually Routes

It is a publish, not a move

The routing key, and the trap in reusing it

Argument or policy, and which one wins

The destination is an ordinary queue

The four reasons, and how each differs

maxlen, and the direction it drops from

TTL, and the ordering constraint that makes it strange

What the broker writes into the message

Dead-lettering is usually described as a place messages go, which is misleading. It is a republish, performed by the broker, through the ordinary routing machinery, and every surprising behaviour follows from that.

It is a publish, not a move

The sentence that explains the rest of the section.

When a message is dead-lettered, the broker publishes it to the exchange named by the queue's x-dead-letter-exchange argument, using either the message's original routing key or the value of x-dead-letter-routing-key if one is set. The exchange then routes it exactly as it would route anything else.

This means a dead-letter exchange with no matching binding discards the message. There is no fallback destination and no error; the publish is unroutable and unroutable messages are dropped. Configuring the exchange without configuring the binding produces precisely the loss the configuration was added to prevent.

It also means the routing is subject to the same rules as any publish. A topic exchange matches patterns, a headers exchange matches arguments, and a message dead-lettered to a fanout exchange goes to every bound queue, which is how one failure produces four copies.

The routing key, and the trap in reusing it

The default is convenient and frequently wrong.

Without x-dead-letter-routing-key, the message keeps the routing key it was published with. If the dead-letter exchange is a different exchange with different bindings, that key may match nothing, which is the discard above.

The worse case is when it matches too well. Pointing a queue's dead-letter exchange at the same exchange the message was originally published to creates a cycle: the message is routed straight back to the queue it just failed in, fails again, and dead-letters again. The broker does not detect this, and the loop runs at the consumer's full rate.

Setting an explicit x-dead-letter-routing-key makes the destination deterministic and is the right default. It also loses information, because the original key is overwritten in the delivery, which is why the original is preserved in x-death and has to be read from there rather than from the delivery's own routing key.

Dead-lettering is a republish through the normal routing machinery, not a move to a special place. A dead-letter exchange with no matching binding therefore discards the message silently, which is exactly the loss the configuration was added to prevent.

Argument or policy, and which one wins

Two places the configuration can live, with different operational properties.

x-dead-letter-exchange can be set as a queue argument at declaration time or as a policy applied by rabbitmqctl set_policy against a name pattern. The argument is fixed for the queue's lifetime, because arguments are immutable and changing one means deleting and redeclaring the queue. A policy can be changed on a running queue.

The argument wins when both are present. A policy that appears to have no effect is usually losing to an argument set by the application's declaration, and the symptom is a configuration change that is visibly applied in the management UI and visibly ignored by the broker.

That makes the choice a real one rather than a style preference. Arguments keep the topology next to the code that depends on it and make it impossible to change without a deploy; policies make the dead-letter target adjustable during an incident, which is occasionally exactly what is needed, and allow one pattern to configure a hundred queues. The mixed case, where some queues get arguments and some get policies, is where environments silently diverge.

The destination is an ordinary queue

Which sounds obvious and is routinely forgotten.

A dead-letter queue has whatever durability, type, length limit and overflow behaviour it was declared with, and none of it is inherited from the source. A transient dead-letter queue behind a durable work queue loses its contents on a broker restart, which is a configuration almost nobody chooses deliberately.

It can also fill. A dead-letter queue with max-length and the default drop-head discards its oldest contents when full, which means the first failures of an incident are the ones deleted and the retained evidence is the least informative. A dead-letter queue with no limit at all is the opposite failure, growing until it trips a memory or disk alarm and blocks publishing across the node.

The configuration that works is a generous limit with reject-publish, plus an alert on depth. The destination is a first-class part of the topology and needs the same sizing attention as the queues feeding it.

The four reasons, and how each differs

Because they are not interchangeable.

rejected means a consumer called basic.reject or basic.nack with requeue false. expired means the message's TTL elapsed, from either the per-message expiration property or the queue's message-ttl. maxlen means the queue was full and x-overflow was set to a dropping policy. delivery_limit means a quorum queue's x-delivery-limit was exceeded.

They differ in who decided. rejected and delivery_limit are statements about the message being unprocessable; expired and maxlen are statements about capacity, and the message may have been perfectly valid. Treating a maxlen dead-letter as a bad message is a misdiagnosis, because the correct response is capacity work rather than inspecting the payload.

The reason appears in the x-death entry, which makes it readable by the consumer of the dead-letter queue, and a handler that branches on it is substantially more useful than one that treats everything as a failure. A dead-letter queue carrying a mixture of all four with no branching is a queue nobody can act on.

maxlen, and the direction it drops from

A detail that determines what survives.

x-overflow takes three values. drop-head, the default, discards the oldest message to make room, so the queue retains the newest. reject-publish refuses the new message instead, which propagates backpressure to the publisher as a basic.nack. reject-publish-dlx refuses it and dead-letters it.

With drop-head, the dead-lettered messages are the oldest ones, which for a work queue is the work that has waited longest and is often the most urgent. With reject-publish, nothing is dead-lettered at all unless the -dlx variant is used, and the publisher learns about the problem, which is usually what you want.

The choice is therefore about which end of the pipeline absorbs the overflow. drop-head protects the broker and silently loses the oldest work; reject-publish protects the data and makes the publisher handle it. Picking drop-head by accepting the default and then being surprised that old messages disappeared is the common path.

TTL, and the ordering constraint that makes it strange

The behaviour that breaks the obvious retry design.

A classic queue evaluates TTL only at the head. A message deep in the queue whose TTL has elapsed is not removed until it reaches the front, so a queue holding a 10-second message behind a 10-minute one does not dead-letter the short one for ten minutes.

Per-message TTL therefore does not give per-message timing in a shared queue. The expiry order is the queue order, which is a head-of-line constraint rather than a scheduler. Queue-level message-ttl is unaffected because every message has the same TTL and the queue order is already the expiry order.

This single fact is why the naive retry ladder does not work, and the next section is largely about it. A queue with mixed per-message TTLs releases them in arrival order, not in deadline order, and a design that assumed otherwise produces delays that are correct for the first message and arbitrary for the rest.

What the broker writes into the message

The record, and its limits.

Dead-lettering prepends an entry to the x-death header array containing queue, reason, exchange, routing-keys, time, count, and for TTL expiry original-expiration. The most recent entry is first, so the array reads as a reverse-chronological path.

The broker also preserves the original expiration property's value in original-expiration and clears the expiration property itself, which prevents the message expiring again immediately in its new queue. That is a deliberate and easily-missed behaviour: a message with a 5-second TTL that is dead-lettered does not carry that TTL forward.

Beyond x-death, the message is unchanged: the same body, the same properties, the same headers. It is a faithful republish, which is what makes a parking-lot queue a usable archive and what makes a retry republish possible at all. The message is the evidence, and nothing in the dead-letter path degrades it.

Two of the four reasons say the message is unprocessable and two say the system ran out of capacity. A dead-letter queue carrying a mixture of all four with no branching on the reason is a queue nobody can act on.
Common Pitfall

A dead-letter exchange with no binding that matches the message's routing key discards the message. There is no fallback destination and no error, because dead-lettering is an ordinary publish and unroutable publishes are dropped. Configuring the exchange without configuring the binding produces exactly the loss the configuration was added to prevent, so declare and test the binding before any reject path exists.