RabbitMQ

01The idea: a post office for messages

Think of RabbitMQ as a post office. A producer drops a letter (message) at the counter (exchange) with an address label (routing key). The post office sorts it into mailboxes (queues) according to rules (bindings). A consumer empties its mailbox and signs for each letter (acknowledgement). If the consumer never signs, the letter goes back to be delivered again. This separation means the producer does not know or care who handles the message, how many workers there are, or whether they are online right now; messages wait safely in the queue.

• Message: a payload (bytes, usually JSON) plus properties like content type, message ID and headers.
• Queue: an ordered buffer that stores messages until a consumer takes them.
• Connection and channel: one TCP connection per process, with lightweight channels inside it for each thread or task.

02Exchanges and routing

Producers never publish directly to a queue; they publish to an exchange. The exchange type decides routing. A direct exchange delivers to queues whose binding key equals the routing key exactly. A topic exchange matches patterns with dots: “order.*” matches “order.created”, and “order.#” matches any depth. A fanout exchange copies every message to all bound queues, ignoring keys. A headers exchange routes by header values. The common design for events is one topic exchange per domain (orders, users) and one queue per consumer purpose (email.order-created, analytics.orders), so each consumer gets its own copy and its own pace.

• Default exchange: a nameless direct exchange that routes to a queue with the same name as the routing key.
• Unroutable messages are dropped unless you publish with mandatory or configure an alternate exchange.

03Queue types: quorum, classic and streams

Quorum queues replicate data across an odd number of nodes using the Raft consensus algorithm; a message is confirmed only after a majority has it, so it survives a node failure. They are the recommended choice for anything important. Classic queues are single-node and non-replicated since RabbitMQ 4.0 removed classic mirrored queues; they are fine for temporary or reproducible work. Streams are an append-only log: consumers read from an offset and messages are not removed on consumption, which enables replay and very high fan-out, similar to Kafka topics. Quorum queues also support a delivery limit, which protects against poison messages that crash consumers repeatedly.

04Publishing reliably

Durable queues and persistent messages survive a broker restart, but they do not tell the producer that the broker got the message. For that, use publisher confirms: the broker acknowledges each message once it is safely stored (for quorum queues, replicated to a majority). If a confirm does not arrive or comes back as a nack, republish. Give each message a unique messageId so consumers can detect duplicates, because retries make delivery at-least-once. Declare exchanges and queues in code or in a definitions file, so environments are reproducible.

1import amqp from "amqplib";
2 
3const conn = await amqp.connect("amqp://your_user:your_password@localhost:5672");
4const ch = await conn.createConfirmChannel();
5 
6await ch.assertExchange("orders", "topic", { durable: true });
7await ch.assertQueue("email.order-created", {
8  durable: true, arguments: { "x-queue-type": "quorum" },
9});
10await ch.bindQueue("email.order-created", "orders", "order.created");
11 
12ch.publish("orders", "order.created", Buffer.from(JSON.stringify({ orderId: 42 })), {
13  persistent: true, contentType: "application/json", messageId: "order-42-created",
14});
15await ch.waitForConfirms(); // throws if the broker rejected the message

05Consuming: acks, prefetch and retries

Consumers should use manual acknowledgements: ack only after the work has succeeded. If the consumer crashes before acking, RabbitMQ redelivers the message to another consumer. Prefetch (basic.qos) limits how many unacknowledged messages one consumer holds; without it, the broker pushes everything to the first consumer and the others sit idle. Start with a prefetch of 10–50 and tune. On failure, decide: requeue for a transient error, or reject without requeue so the message goes to a dead letter exchange (DLX) for inspection or delayed retry. Never ack before processing, and make handlers idempotent.

1await ch.prefetch(20);
2 
3await ch.consume("email.order-created", async (msg) => {
4  if (!msg) return;
5  try {
6    const event = JSON.parse(msg.content.toString());
7    await sendOrderEmail(event.orderId); // must be idempotent
8    ch.ack(msg);
9  } catch (err) {
10    // false, false = do not requeue: route to the dead letter exchange
11    ch.nack(msg, false, false);
12  }
13}, { noAck: false });

06Dead letters, policies and running it

A dead letter exchange receives messages that were rejected, expired by TTL or exceeded the delivery limit. Configure it with a policy rather than hard-coding queue arguments, because policies can be changed without redeclaring queues. A classic retry pattern routes failed messages to a retry queue with a TTL whose own DLX points back to the work exchange, giving delayed retries. Run RabbitMQ as a three-node cluster for quorum queues, enable the management plugin for the UI and HTTP API, expose metrics with the Prometheus plugin, and use TLS and per-application users with limited vhost permissions. Delete the default guest user, which only works from localhost anyway.

1# Dead-letter failed messages and cap redeliveries for all email.* queues
2rabbitmqctl set_policy email-dlx "^email\." \
3  '{"dead-letter-exchange":"dlx","delivery-limit":5}' --apply-to quorum_queues
4 
5rabbitmqctl add_vhost shop
6rabbitmqctl add_user orders_app 'use-a-strong-password'
7rabbitmqctl set_permissions -p shop orders_app '^orders.*' '^orders.*' '^(orders|email).*'

07Quorum queues under the hood

A quorum queue is a Raft group: one leader and followers on other nodes, typically three or five members. Publishes go to the leader, which appends them to its log and replicates; once a majority has written the entry to disk, the leader confirms it to the publisher. If the leader's node fails, the remaining majority elects a new leader within seconds and consumers continue. Because every message is replicated and fsynced, quorum queues use more disk I/O and are best for queues that are drained regularly, not for millions of messages parked for days. They track redelivery counts, enforce a delivery limit (a default exists since 4.0), and support message priorities since 4.0. Very long queues should become streams.

• Use three members for most queues; five tolerate two failures at more cost.
• Members are placed across cluster nodes; keep nodes in separate availability zones.

08Flow control, alarms and back-pressure

RabbitMQ protects itself. When a node's memory use passes the high watermark (by default a fraction of RAM) or free disk drops below the disk limit, it raises an alarm and blocks all publishing connections cluster-wide until resources recover. Consumers keep working, which is how the system drains. Credit-based flow control also slows fast publishers that outrun queue processes. In practice, alarms are a symptom of a queue growing faster than it is consumed. Set max-length or TTL policies on queues that may grow unbounded, alert on queue depth growth rate, scale consumers, and give the broker fast disks with headroom. Clients should handle the connection.blocked notification and stop or buffer publishing.

09Retries with backoff and poison messages

Requeueing immediately retries too fast and can loop a poison message. A robust pattern uses dead lettering for delays: on failure, the consumer rejects the message, which goes via a DLX to a retry queue with a message TTL (for example 30 seconds) and no consumers; when the TTL expires, that queue dead-letters it back to the main exchange. Use several retry queues for exponential steps (30 s, 5 min, 30 min) and count attempts in a header or rely on the x-delivery-count header in quorum queues. After the last attempt, route to a parking queue that a human inspects. A delayed message exchange plugin exists too, but it is not replicated, so TTL queues are safer for critical work.

1# Retry queue: holds failed messages for 30 s, then sends them back
2rabbitmqctl set_policy retry-30s "^retry\.30s$" \
3  '{"message-ttl":30000,"dead-letter-exchange":"orders"}' --apply-to queues
4 
5# Main work queues dead-letter into the retry exchange
6rabbitmqctl set_policy work-dlx "^email\." \
7  '{"dead-letter-exchange":"retry","delivery-limit":10}' --apply-to quorum_queues

10Exactly-once is a myth: idempotency and the outbox

Networks fail between “processed” and “acked”, and between “published” and “confirmed”, so any reliable system is at-least-once. Two patterns make that safe. The transactional outbox on the producer side: write the business change and an outbox row in the same database transaction, and let a relay publish outbox rows with confirms and mark them sent. This prevents “order saved but event lost” and “event sent but order rolled back”. The idempotent consumer on the receiving side: store processed message IDs (or use natural keys and upserts) in the same transaction as the side effect, and skip duplicates. With both, the system behaves as if delivery were exactly once.