Queues, streams and delivery guarantees

Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.

Concept: Message queues and event streams · also covers Kafka, Batch and real-time workloads, Distributed systems · Lesson 7 of 31 in System design · version 1

Practice this

On this page

Messaging decouples the service that produces work from the services that process it, in time and in capacity. Queues hand each message to one worker and forget it, while streams keep an ordered, replayable log that many independent consumers read at their own pace. The guarantees that matter are ordering, durability and how many times a message can be processed, and each is a deliberate configuration choice.

Read more

How it is done in practice

Teams decide first whether the payload is a command for one worker or an event many systems will read, then pick a queue service or a Kafka topic accordingly. For Kafka they choose the record key from the ordering requirement, set a replication factor of three with acks from all in-sync replicas, enable the idempotent producer and agree on a schema with compatibility rules. Consumers are written to be idempotent and commit offsets after processing. Dashboards show consumer lag and dead letter volume per group, with alerts on lag growth, and runbooks describe how to replay a topic or reset offsets.

Technologies

Apache KafkaRabbitMQAmazon SQSAmazon KinesisGoogle Pub/SubApache PulsarNATSAzure Service BusSchema Registry

Diagram

flowchart LR
    P1[Producer] -->|key hash| T0[Partition 0]
    P1 -->|key hash| T1[Partition 1]
    P1 -->|key hash| T2[Partition 2]
    T0 --> A1[Group A consumer 1]
    T1 --> A1
    T2 --> A2[Group A consumer 2]
    T0 --> B1[Group B consumer]
    T1 --> B1
    T2 --> B1
    A2 -->|after retries| D[Dead letter topic]
flowchart LR
    P1[Producer] -->|key hash| T0[Partition 0]
    P1 -->|key hash| T1[Partition 1]
    P1 -->|key hash| T2[Partition 2]
    T0 --> A1[Group A consumer 1]
    T1 --> A1
    T2 --> A2[Group A consumer 2]
    T0 --> B1[Group B consumer]
    T1 --> B1
    T2 --> B1
    A2 -->|after retries| D[Dead letter topic]

Code

bash
# create a topic with 6 partitions, then inspect consumer lag
bin/kafka-topics.sh --create --topic orders --partitions 6 \
  --replication-factor 3 --bootstrap-server HOST:9092
bin/kafka-consumer-groups.sh --describe --group billing \
  --bootstrap-server HOST:9092
python
# at-least-once consumer with an idempotent handler
for msg in consumer:                      # auto commit disabled
    event_id = msg.headers_dict['event-id']
    with db.transaction() as tx:
        if tx.seen(event_id):             # duplicate delivery
            continue
        apply_event(tx, msg.value)
        tx.mark_seen(event_id)
    consumer.commit()                     # commit after processing

Subtopics

Queues versus logs

Two models

A message queue holds tasks until a worker takes one, processes it and acknowledges it, after which the broker deletes it. Several workers can pull from the same queue, which is the competing consumers pattern: throughput scales by adding workers, and the queue absorbs bursts so the producer is not blocked. A log based stream, as in Kafka, instead appends events to a durable ordered log and keeps them for a retention period regardless of consumption. Consumers track their own position, so new consumers can be added later and can replay history.

When to use which

Choose a queue for work distribution: jobs, emails, image processing, where each task is independent, per message acknowledgement and redelivery matter, and order usually does not. Choose a stream when events are facts that several systems need, when order per entity matters, or when replay is valuable for rebuilding state and backfilling a new consumer. A stream can act as a queue through consumer groups, but per message retry and delay are more awkward there because position is a single offset per partition.

Partitions, offsets and consumer groups

How a stream scales

A Kafka topic is split into partitions, each an ordered append only log stored on brokers and replicated for fault tolerance. Each record gets an offset, its position in the partition. Producers choose the partition, normally by hashing the record key, so all events with the same key land in the same partition and stay in order. Order is guaranteed within a partition only, never across the topic. The key is therefore a design decision: use the entity whose events must be ordered, such as an order id or account id.

Consumer groups

Consumers that share a group id divide the partitions among themselves, so each partition is read by exactly one member of the group at a time. That gives parallelism up to the partition count; extra consumers sit idle. Different groups read the same topic independently, each with its own committed offsets. When a member joins or dies, the group rebalances and partitions are reassigned, and the new owner resumes from the last committed offset. Partition count caps parallelism and is hard to change later without affecting key to partition mapping.

At-most-once, at-least-once and exactly-once

The three guarantees

The guarantee comes from the order of processing and acknowledging. Acknowledge or commit the offset first, then process: a crash in between loses the message, giving at-most-once. Process first, then acknowledge: a crash in between causes redelivery, giving at-least-once with possible duplicates. At-least-once is the practical default, because lost messages are usually worse than duplicates. On the producer side the same choice exists: a producer that retries after a timeout may write the same message twice unless the broker can detect the duplicate.

Exactly-once in Kafka

Kafka offers two mechanisms. The idempotent producer attaches a producer id and sequence numbers so the broker discards retried duplicates within a partition. Transactions let a producer write to several partitions and commit consumer offsets atomically, so a read, process, write loop inside Kafka takes effect once; consumers must read with isolation level read committed. This covers Kafka to Kafka flows. Any side effect outside Kafka, such as a database write or an API call, still needs an idempotent consumer, typically by recording a message id or using an upsert.

Backpressure, lag, retries and dead letters

Operating consumers

A broker hides overload instead of removing it: if consumers are slower than producers, the backlog grows. On a queue the signal is queue depth and message age; on a stream it is consumer lag, the gap between the latest offset and the committed offset. Scale consumers on these signals, and set retention so that lag cannot outrun it. A message that always fails, a poison message, will block a partition or cycle forever in a queue. Retry a bounded number of times with backoff, then move it to a dead letter queue or topic with the error attached, alert on it and provide a way to replay after the fix.

Sources

Related lessons

Practice this
CachingLoad balancing and service discovery