Practice: Webhooks and event-driven integrations with Kafka
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Kafka · Kafka is the durable, replayable log the lesson is about, and its topics, partitions, offsets and consumer groups can all be exercised on a laptop with the scripts that ship in the download. · Alternatives: Redpanda (Kafka-compatible API, single binary, no JVM), RabbitMQ (queue broker that routes and deletes on ack, no replayable log), NATS (lightweight messaging, persistence only through JetStream), AWS SQS and Amazon EventBridge (managed queue and event bus, cloud only, little or no replay), Amazon MSK (the same Kafka run as a managed service).
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Skim the Getting Started landing page to see which guides exist and in what order they are meant to be used.
Knowing where the introduction, quickstart and use cases sit saves hunting for them later in the path.
Explained in: Getting Started | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: You can name the guide you will open for the first run and the one that explains the concepts.
2. Install a supported Java runtime, then download and extract the latest Kafka release into its own directory.
Every command line tool used in this path ships inside that release, and the broker needs Java to start.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The extracted directory lists the bin and config folders and the Java version check prints a supported version.
3. Format the log directory and start a single local broker from the downloaded files.
A single broker in KRaft mode is the smallest working Kafka and is enough for most of the steps.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The broker log shows the server has started and the terminal stays attached without errors.
Steps
1. Create a topic named webhook-raw on the local broker and describe it.
A topic is where every event in this path will live, and describing it shows the partition count and leader you will reason about later.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The describe output lists webhook-raw with its partition count, leader and replicas.
2. Write a handful of events into webhook-raw with the console producer, one event per line.
Producing by hand shows that a write is an append to a log, not a call to a receiver.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The producer accepts each line without error and you exit it cleanly.
3. Run the console consumer from the beginning of webhook-raw, stop it, then run it again from the beginning.
Reading the same events twice proves the log is durable and replayable, which is the property polling and plain webhooks lack.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: Both runs print the same events in the same order.
4. Read the concepts section and write down in your own words what a topic, a partition, an offset, an event key and a consumer group are.
Every delivery and ordering question in a customer deployment is answered in these terms, so they must be exact before you go on.
Explained in: Introduction | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: You can explain why events with the same key land in the same partition and why ordering holds only inside one partition.
5. Create a three-partition topic named webhook-events, start two console consumers with the same group name, then produce events and stop one consumer.
Watching partitions get split and then reassigned is the fastest way to understand how a consumer group scales and recovers.
Explained in: Introduction | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: Each event is printed by exactly one of the two consumers, and the surviving consumer takes over all partitions after the other stops.
6. Write a small local HTTP receiver that verifies a shared-secret signature on each request, produces the raw body to webhook-raw with the producer API, and only then returns a success status, leaving all processing to a separate consumer.
Verify, acknowledge, defer keeps the sender from timing out and retrying, and putting the raw payload on the log first means nothing is lost if processing fails.
Explained in: Introduction | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: A signed test request returns success quickly and appears on webhook-raw, while an unsigned one is rejected and never reaches the topic.
7. Produce the same event id twice and two updates for one entity out of order, keyed by entity id, then write a consumer that skips seen ids and rebuilds current state by replaying the topic from the first offset.
Webhook senders retry and reorder, so the consumer must be idempotent, and replaying the log is how you reconcile after a bug or a missed delivery.
Explained in: Uses | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The rebuilt state shows one result per event id and is identical after a second full replay.
8. Configure your consumer to commit offsets only after processing, stop it in the middle of a batch and restart it, then add a schema check that sends any event failing validation to a dead letter topic instead of retrying it forever.
Committing after processing gives at-least-once delivery with visible redelivery, and a poison message that is not diverted blocks its whole partition.
Explained in: Uses | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: Restarting reprocesses the uncommitted events, and a malformed event shows up on the dead letter topic while the valid events after it keep flowing.
9. Run Kafka Connect in standalone mode with the file source and file sink connectors as a stand-in for a bridge between an outside system and the event backbone.
In a customer deployment the bridge is usually a connector and configuration, not custom code, and this is the smallest example of that pattern.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: Lines appended to the source file appear on the connector topic and in the sink file without restarting anything.
10. Run the Kafka Streams word count demo against an input topic and watch the output topic update as you produce more events.
Downstream teams consume the bridged webhook events as continuous transformations, so you should see one topic feeding another through a running application.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The output topic prints updated counts each time you produce a new input line.
11. Reproduce the whole path from a clean state by terminating the environment, deleting the local data, and starting the broker again from the pinned JVM Docker image tag.
A pinned image and a clean start are what make the setup repeatable for a teammate or a CI job instead of something that only works on your laptop.
Explained in: Quickstart | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: After the reset, the topics are gone, and recreating them and rerunning your receiver and consumer gives the same results as before.
12. Configure a three-broker local cluster in containers as the substitute for a paid managed service, create webhook-raw with a replication factor of three, then stop one broker while the receiver and consumer are running.
Replication across brokers is what makes the log safe to depend on in production, and a broker failure is the first thing a customer will ask about.
Explained in: Introduction | Apache Kafka (kafka.apache.org), checked 2026-10-11
Done when: The describe output shows three replicas per partition, and produce and consume continue with a new leader after one broker is stopped.