Event-driven systems: Kafka, RabbitMQ and queues

01Events, commands and brokers

A command asks someone to do something: 'send this email', 'charge this card'. It has one intended handler and can be rejected. An event states a fact that already happened: 'order 42 was paid'. It has no opinion about who listens; zero or ten consumers may react. Mixing these up causes trouble: an 'event' named SendEmail couples the producer to one consumer. A message broker is the post office between them. Producers publish messages to it, it stores them durably, and consumers receive them at their own pace. If a consumer is down, messages wait. If traffic spikes, the backlog grows instead of the database melting. The price is asynchrony: the producer no longer knows when, or whether, the work is done, so you need monitoring of backlogs and failures.

• Command: imperative, one handler, may fail ('ReserveStock').
• Event: past tense, many handlers, immutable fact ('StockReserved').
• Broker: durable buffer and router between producers and consumers.

02Kafka: topics, partitions, offsets and consumer groups

Kafka is a distributed append-only log. Messages are written to a topic, which is split into partitions; each partition is an ordered sequence where every message gets an offset (its position number). Messages with the same key always go to the same partition, so ordering is guaranteed per key (for example, per order ID), never across the whole topic. Consumers join a consumer group; Kafka assigns each partition to exactly one consumer in the group, so the number of partitions caps parallelism. Each group tracks its own committed offsets, which means many groups (billing, analytics, search) read the same topic independently. Messages are not deleted when read: they stay for a retention period or size, or are compacted to the latest value per key. Since Kafka 4.0, clusters run on KRaft without ZooKeeper, and Kafka 4.2 added production-ready share groups for queue-style consumption.

1# Create a topic: 6 partitions, 3 replicas, 7-day retention
2kafka-topics.sh --bootstrap-server localhost:9092 --create \
3  --topic orders.v1 --partitions 6 --replication-factor 3 \
4  --config retention.ms=604800000 --config min.insync.replicas=2
5 
6# Produce keyed messages (same key -> same partition -> ordered)
7kafka-console-producer.sh --bootstrap-server localhost:9092 \
8  --topic orders.v1 --property parse.key=true --property key.separator=:
9 
10# Inspect a consumer group: current offset, end offset and lag per partition
11kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
12  --describe --group billing-service

03RabbitMQ and Redis queues

RabbitMQ is a message router. Producers publish to an exchange, which routes messages to queues through bindings: direct (exact routing key), topic (patterns like 'order.*'), fanout (every bound queue) or headers. Consumers take messages from queues and acknowledge them; unacknowledged messages are redelivered, and rejected ones can go to a dead-letter exchange. Once acknowledged, a message is gone, so RabbitMQ is about delivering work, not keeping history (its stream queues add a log-like mode when you need replay). It shines at per-message routing, priorities and request-reply. Redis-based queues such as BullMQ (Node) or Laravel queues with a Redis driver are the lightest option: jobs with retries, backoff, delays, rate limits, scheduled and repeatable jobs, all on the Redis you probably already run. Their limit is durability and scale: Redis keeps data in memory, so configure persistence and treat the queue as a job runner, not a permanent event store.

04Delivery guarantees in practice

Every broker offers some mix of three guarantees. At-most-once: the consumer commits before processing, so a crash loses the message but never duplicates it; acceptable for metrics, rarely for business data. At-least-once: the consumer processes, then acknowledges; a crash between the two means redelivery, so duplicates happen. This is the realistic default for most systems. Exactly-once: each message affects the result once. Kafka supports it inside Kafka, using idempotent producers and transactions with consumers reading only committed data, for read-process-write pipelines between Kafka topics. As soon as a side effect leaves Kafka (a database write, an email, a card charge), you are back to at-least-once and must make the consumer idempotent: store processed message IDs, use unique constraints or upserts, and design handlers so that running them twice changes nothing.

• Producer side: acks=all and idempotence on (the default in modern Kafka clients).
• Consumer side: commit offsets after successful processing, not before.
• Poison messages: retry with backoff, then move to a dead-letter topic or queue.

05Schemas, versioning and choosing a tool

Events are a public contract, often read by teams you never meet, so treat their shape like an API. Define schemas in Avro, Protobuf or JSON Schema, store them in a schema registry, and enforce compatibility rules: add optional fields freely, never remove or repurpose a field consumers rely on. For breaking changes, publish a new event type or topic version (orders.v2) and run both until consumers migrate. Include an event ID, type, version, timestamp and the entity key in every message. To choose: Kafka for high-volume streams, long retention, replay and many consumer groups; RabbitMQ for flexible routing, task distribution and per-message acknowledgement; Redis queues like BullMQ for application background jobs with retries and scheduling; managed services (SQS, SNS, Pub/Sub, managed Kafka) when you prefer not to operate brokers yourself.

06Durability inside Kafka: replicas, ISR and leader election

Every partition has one leader replica that takes writes and followers that copy it. The in-sync replica set (ISR) is the leader plus followers that are caught up within replica.lag.time.max.ms. With acks=all, the producer gets a success only when every member of the current ISR has the record; min.insync.replicas sets how small the ISR may shrink before writes are refused with NotEnoughReplicas. The common durable setup is replication factor 3, min.insync.replicas 2 and acks=all: one broker can fail with no data loss and no write outage. Keep unclean.leader.election.enable at its default of false; enabling it lets an out-of-sync replica become leader and silently drop acknowledged messages in exchange for availability. Spread replicas across racks or availability zones with broker.rack so one zone outage cannot take every copy.

• RF 3 + min.insync.replicas 2 + acks=all: survives one broker loss safely.
• Producer idempotence and acks=all are client defaults since Kafka 3.0.
• Unclean election trades acknowledged data for availability; leave it off.

07Rebalancing, the new consumer protocol and processing stalls

When consumers join, leave or crash, partitions are reassigned in a rebalance. The classic protocol could pause the whole group; cooperative-sticky assignment reduced that, and the next-generation protocol (KIP-848), generally available since Kafka 4.0, moves assignment to the broker-side group coordinator and rebalances incrementally. Clients opt in with group.protocol=consumer; groups convert during a rolling upgrade. Two timeouts matter. session.timeout.ms detects a dead consumer through heartbeats. max.poll.interval.ms detects a stuck one: if your handler takes longer than this between poll calls, the consumer is evicted, its partitions move and the uncommitted batch is processed again elsewhere. Slow handlers therefore cause duplicate work and rebalance storms. Lower max.poll.records, move long work to a separate queue, or pause partitions while processing.

08Compaction, retention and replay at scale

Retention can be time or size based (delete policy) or key based (compact policy). Compaction keeps at least the latest record per key and removes older ones in the background, which turns a topic into a durable table of current state: customer profiles, product prices, configuration. A record with a null value is a tombstone; it marks the key for deletion and is itself removed after delete.retention.ms, so consumers that fall further behind than that can miss deletions. Combining both policies keeps the latest value per key and still expires very old keys. Replay is the superpower: a new consumer group starting at the earliest offset rebuilds a read model from history. Plan for it: rebuilds can read terabytes, so throttle them, give them their own group, and make sure retention is longer than your worst realistic recovery time.

1# A compacted topic holding the latest price per product
2kafka-topics.sh --bootstrap-server localhost:9092 --create \
3  --topic prices.v1 --partitions 12 --replication-factor 3 \
4  --config cleanup.policy=compact \
5  --config min.compaction.lag.ms=3600000 \
6  --config delete.retention.ms=86400000
7 
8# Rebuild a read model: new group, start from the beginning
9kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
10  --group search-indexer-v2 --topic prices.v1 \
11  --reset-offsets --to-earliest --execute

09Partition sizing, key skew and throughput tuning

Partitions are the unit of parallelism and ordering, so size them for the consumer parallelism you expect in a year or two, not today: adding partitions later remaps keys and breaks per-key ordering during the change. Too many partitions also cost: more open files, longer leader elections and more metadata. A key with far more traffic than others creates a hot partition that one consumer must handle alone. Fix skew by choosing a finer key (order ID rather than customer ID) when ordering only matters per order, or by salting a known hot key across a few sub-keys and merging downstream. Records without a key are spread by the sticky partitioner, which fills one batch before moving on. On the producer, linger.ms and batch.size trade a few milliseconds of latency for much higher throughput, and compression (lz4 or zstd) cuts network and disk use.