RabbitMQ dead-lettering, TTL and counted retries: where do failed jobs go? Intermediate

The problem

A consumer sends an email, the provider returns an error, and the consumer nacks with requeue=true. RabbitMQ redelivers immediately and the error happens again. Within seconds, one job eats most of the CPU and logs, and valid jobs slow down.

Switch to requeue=false and the loop stops, but if the queue has no dead-letter exchange (DLX), the message is dropped for good. Both approaches lack a policy: which errors to retry, how long, how many times, and where to store the job when retries run out.

Overview: why a message leaves a queue

A DLX is a redirection mechanism: when a message leaves a queue for one of the four reasons below, RabbitMQ republishes it to the DLX. It does not decide whether to retry.

reason	When it happens	Notes
rejected	basic.reject or basic.nack with requeue=false	Initiated by the consumer
expired	Message TTL expired	Quorum and classic handle it when the message reaches the head of the queue
maxlen	The queue exceeds its length limit	With overflow drop-head (the oldest message at the head) or reject-publish-dlx (the rejected publish); reject-publish does not dead-letter
delivery_limit	A quorum queue redelivers beyond x-delivery-limit	Default 20 since RabbitMQ 4.0

When a whole queue expires (queue TTL), its messages are not dead-lettered.

The consumer rejects bad-job; the broker routes it to the dead queue

Enlarge diagram

Main queue → Consumerbad-job

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The main queue delivers the message to the consumer; the message is now unacked.

Declaring it correctly

The docs recommend configuring with a policy instead of hardcoding arguments (if both set a DLX, the argument wins).

# Policy for the main queue: DLX + routing key
rabbitmqctl set_policy dlx-main '^main\.' \
  '{"dead-letter-exchange":"dlx","dead-letter-routing-key":"dead"}' \
  --apply-to queues

Key names checked against the documentation:

Purpose	Policy key	Queue argument
DLX	dead-letter-exchange	x-dead-letter-exchange
Routing key when dead-lettering	dead-letter-routing-key	x-dead-letter-routing-key
TTL for every message in the queue	message-ttl (ms)	x-message-ttl
TTL for a single message	expiration property (ms string)
Delete an unused queue	expires	x-expires
Dead-letter strategy (quorum)	dead-letter-strategy	x-dead-letter-strategy
Redelivery limit (quorum)	delivery-limit	x-delivery-limit

If both a queue TTL and a message TTL are set, the lower value applies. x-expires is the TTL of the whole queue when unused, not a message TTL; putting it in the wrong place deletes the topology.

TTL is not a precise clock

An expired message is never delivered to a consumer, but the broker only dead-letters it when it reaches the head of the queue. An expired message behind an unexpired one can sit there until its turn. So “retry after exactly 10 seconds” is only an approximation.

Design consequence: use one retry queue per delay level (e.g. 10s, 1m, 10m); do not mix different TTLs in the same queue.

Counted retries

The model: the main queue has a DLX pointing to the retry queue; the retry queue has a TTL and a DLX pointing back to main; the dead queue is the end point.

The main, retry (TTL), main loop and the stop at the dead queue

Enlarge diagram

main.queue → Consumerjob

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

Attempt 1: the consumer receives the job and the provider fails with 503. There is no x-death yet, so the retry count is 0.

The counter comes from the x-death header. Each entry contains queue, reason, count, exchange, routing-keys, time; count is the number of times the message was dead-lettered from that queue for that reason. The consumer reads the count of the (queue=retry, reason=expired) entry:

// Pseudocode: number of retries so far
const deaths = msg.properties.headers?.['x-death'] ?? [];
const retries = deaths.find(d => d.queue === 'retry' && d.reason === 'expired')?.count ?? 0;
if (retries >= 3) { publishToDead(msg); channel.ack(msg); }   // out of retries
else channel.nack(msg, false, false);                          // to retry.queue

Why this loop is not cut: RabbitMQ detects dead-letter cycles and drops a message only if the cycle contains no rejected step. Here the consumer nacks each round, so the cycle is valid.

The model above is based on the documentation; the repo has no runnable demo for the retry loop yet. Only the nack-to-dead-queue part below has been run.

Classify errors before retrying:

Error type	Action
Transient (503, network timeout)	Nack, go through the counted retry loop
Permanent (payload with the wrong schema)	Publish straight to the dead queue, no retry
Business (“resource already deleted”)	Ack and record it per a separate policy

Safe dead-lettering: at-least-once

By default, dead-lettering is at-most-once: it uses no confirms, so messages can be lost if the target is unavailable. Quorum queues have an at-least-once mode when all of the following hold (checked against the quorum queues docs):

1. The source queue is a quorum queue.
2. Policy dead-letter-strategy = at-least-once (default at-most-once).
3. overflow = reject-publish (default drop-head; with drop-head this mode falls back to at-most-once).
4. dead-letter-exchange is set; the routing key is optional.
5. The stream_queue feature flag is enabled (check with rabbitmqctl list_feature_flags).
6. Setting max-length or max-length-bytes is recommended.

rabbitmqctl set_policy dlx-safe '^orders\.' \
  '{"dead-letter-exchange":"dlx","dead-letter-strategy":"at-least-once","overflow":"reject-publish","max-length":100000}' \
  --apply-to quorum_queues

In exchange, the source queue holds messages until the target confirms, costing extra resources. You still need to monitor whether the dead queue keeps growing.

Poison messages and x-delivery-limit

Since RabbitMQ 4.0, quorum queues have x-delivery-limit with a default of 20. A message redelivered more than 20 times (consumer died or requeue=true) is dead-lettered with reason delivery_limit, or dropped if there is no DLX.

Setting -1 disables the limit, but the docs do not recommend it. Since 4.3 the limit is compared with delivery-count instead of acquired-count.

This is the last safety net for requeue=true loops, not a replacement for counted retries: 20 back-to-back redeliveries are still 20 hot loops.

Demo that was run

docker compose -f examples/docker-compose.yml up -d rabbitmq
npx --prefix examples ts-node examples/src/rabbitmq-dlx/dlx-demo.ts

Output on rabbitmq:4-management-alpine: dead queue: bad-job and x-first-death-reason: rejected. The demo only verifies the nack, DLX, dead queue path. It does not test TTL, quorum queues, at-least-once or a broker restart.

Try changing the dead-letter routing key to a key with no binding: the dead queue stays empty. The real topology decides where messages go, not the name “DLX”.

Real-world example

Partner webhooks: three retry tiers

A service sends webhooks to partners, about 5,000 jobs/hour.

• 503 or timeout: a retry loop with two queues (10 seconds, 5 minutes), at most 3 rounds counted with x-death.
• Wrong-schema payload: straight to the dead queue, with a trace ID.
• Out of retries: dead queue; alert when the oldest message is older than 1 hour.
• Runbook: inspect the PII-masked payload, fix the bug, replay with limits, keep the event ID so downstream dedupes.

When to use / not use

• Use DLX + retry queues: transient errors that need backoff, without a consumer sleep holding an unacked delivery.
• Go straight to the dead queue: permanent errors.
• Do not use TTL for tight timing SLAs: expiry is only approximate; use a scheduler.
• Need large-scale topic-based retries and long-term replay: consider streams or Kafka (see Kafka vs. RabbitMQ).

Production checklist

1. Every job queue has a DLX, and the dead queue has an owner, retention and a runbook.
2. Test bindings with a real message, not just by reading config.
3. Important quorum queues: at-least-once + reject-publish + max-length.
4. Do not use requeue=true as a retry strategy.
5. Counted retries via x-death or an application header, with a threshold and an end point.
6. One retry queue per delay level.
7. Replays keep the original event ID; consumers are idempotent.
8. Monitoring: dead queue depth and age, delivery_limit rate.

Common mistakes

Common pitfall

Nacking with requeue=true for every error. Permanent errors loop hot until they hit the delivery limit. Classify errors and back off on transient ones.

Common pitfall

Believing drop-head is still safe. With the default drop-head overflow, at-least-once dead-lettering does not work. Set reject-publish.

Common pitfall

Confusing queue TTL with message TTL. x-expires deletes the whole queue; to delay retries, use x-message-ttl.

Summary

Key takeaways

• A DLX redirects messages for rejected, expired, maxlen, delivery_limit.
• Quorum queues in 4.x have x-delivery-limit defaulting to 20; at-least-once requires quorum + reject-publish + DLX.
• TTL only dead-letters once a message reaches the head of the queue; it is not a timer.
• Retries need a counter (x-death), a threshold and a stop at the dead queue.

Self-check: (1) Does a nack with requeue=true reach the DLX? (2) Why is the main, retry, main loop not cut by cycle detection? (3) What does drop-head overflow do to at-least-once?

References

• Dead Letter Exchanges — RabbitMQ
• Time-To-Live and Expiration — RabbitMQ
• Quorum Queues — RabbitMQ