Practice: Queues, streams and delivery guarantees
Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.
Main tool: Apache Kafka · Apache Kafka runs as a single local broker and exposes the log, partitions, offsets and consumer groups directly through its console tools, so every delivery guarantee in this lesson can be observed by hand. · Alternatives: RabbitMQ and Amazon SQS are queues that hand each message to one worker and then delete it, Amazon Kinesis and Google Pub/Sub are managed cloud streams with less to operate but no laptop install, Apache Pulsar combines queue and log semantics with tiered storage, NATS is a lighter broker with optional persistence, and Azure Service Bus is a managed queue with built-in dead lettering.
Every step links into the tool's own documentation; links were checked on 2026-10-13 · version 1
Setup (one time)
1. Install a current Java runtime, then download and extract the latest Kafka release into a new empty directory.
The broker and all the console tools used in this path ship in that one archive and run on the JVM.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The extracted Kafka directory exists and its bin folder lists the topic, console producer and console consumer tools.
2. Generate a cluster id, format the log directory and start a single local broker from the downloaded files.
A single local broker is enough to practise every concept in this lesson without any cloud account.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The broker log prints that the Kafka server started and the terminal stays attached without errors.
3. Read the Getting Started landing page to see which guides exist and where the quickstart sits among them.
Knowing the layout of the documentation saves time when a later step sends you back for an option or a concept.
Explained in: Getting Started | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: You can name the getting started guides and say which one each later step of this path draws on.
Steps
1. Create a first topic named quickstart-events and describe it with the topic tool.
A topic is the named log that every producer and consumer in this path will share, so it must exist before anything else.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The describe output shows the topic name, its partition count, its replication factor and a leader for each partition.
2. Write a handful of events into the topic with the console producer, one event per line.
Producing by hand shows that the producer only appends to the log and knows nothing about who will read it.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: Each line you type is accepted without error and the producer waits for the next one.
3. Run the console consumer with the from-beginning option in a second terminal and keep it open while you produce more events.
Seeing old and new events arrive in order is the first proof that the broker stored them instead of handing them off and forgetting them.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The consumer prints every event written so far in order, and each new line typed in the producer appears within a second.
4. Read the main concepts section and map events, topics, partitions, producers and consumers onto what you just ran.
The object model explains why ordering holds only inside one partition and why consumers track a position instead of deleting messages.
Explained in: Introduction | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: You can explain in your own words how an event key selects a partition and what ordering a consumer of that partition is promised.
5. Create a second topic with three partitions using the partitions option of the topic tool, then describe it.
Partitions are the unit of parallelism and of ordering, so you need a topic with more than one to see how work is spread.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The describe output lists three partitions, each with its own leader.
6. Start two console consumers that share one group name on the three-partition topic, then a third consumer with a different group name, and produce events.
Consumers in one group split the partitions between them like workers on a queue, while a separate group reads the whole log independently from its own offsets.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: Each event appears in exactly one of the two same-group consumers and also in the consumer of the other group.
7. Compare queue and log behaviour by stopping every consumer, then starting a new one with a fresh group name and the from-beginning option on the same topic.
A traditional queue would be empty after the first read, whereas the log keeps events so a new subscriber can replay them, which is the central trade-off of this lesson.
Explained in: Uses | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The new consumer prints all events that earlier consumers had already processed, and you can state which use cases need that replay.
8. Reproduce a duplicate delivery by stopping a consumer partway through, restarting it from the beginning, and counting how often each event was printed.
Where the consumer position is recorded relative to the work decides between at-most-once, at-least-once and exactly-once, and replaying makes the at-least-once case visible.
Explained in: Introduction | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: You have a list of events that were printed twice and can say what the consumer would need to change to get at-most-once or exactly-once processing instead.
9. Write a burst of several hundred events while every consumer of one group is stopped, then restart one consumer and watch it catch up.
The gap between the newest event and the group position is consumer lag, and a broker that absorbs the burst is how a log gives backpressure relief to a slow consumer.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The restarted consumer prints the whole backlog in order without any event lost, and you can say how the backlog would be measured as lag.
10. Create a retry topic and a dead letter topic, then move one event you declare as failing from the main topic into each with the console tools.
Kafka has no built-in dead letter queue for plain consumers, so retries and poison messages are handled by routing them to further topics that you design.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: All three topics appear in the topic list and a consumer on the dead letter topic prints the failed event.
11. Terminate the environment, delete its local data, and start the broker again from the JVM based Docker image pinned to one exact version, then repeat the topic, produce and consume steps.
A pinned container image makes the whole exercise reproducible on any laptop or CI runner, which is the closest local stand-in for a managed deployment.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-13
Done when: The container reports the broker started, the old topics are gone, and the recreated topic accepts and returns events as before.