Practice: Outbox and change data capture
Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.
Main tool: Debezium · Debezium reads the database's own commit log and ships an outbox event router, so one tool lets you practise log based capture, outbox routing and rebuilding derived data on a laptop. · Alternatives: Polling outbox publisher in the application (simple, but adds latency and load, and ordering is harder to guarantee); Kafka Connect JDBC source (query based, misses deletes and intermediate states); Maxwell or Canal (MySQL binlog only); DynamoDB Streams (managed change stream tied to one cloud database); PostgreSQL logical replication consumed directly (no connector framework, you write offset handling yourself).
Every step links into the tool's own documentation; links were checked on 2026-10-13 · version 1
Setup (one time)
1. Read what Debezium is and sketch the pipeline you will build: PostgreSQL, Kafka, Kafka Connect with Debezium, and one consumer.
A clear picture of the moving parts makes every later step easier to debug. It also fixes where the log, the connector and the topics sit.
Explained in: Debezium Documentation :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You have a one-page diagram naming each container and the direction the data flows.
2. Install a container runtime and start Kafka plus a Kafka Connect container with the Debezium connector plugins on its plugin path.
Debezium connectors run inside Kafka Connect, so the plugins must be visible to the worker before any connector can be registered.
Explained in: Installing Debezium Kafka Connect Connectors :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: The Connect REST interface lists the Debezium PostgreSQL connector class among its installed plugins.
3. Start a PostgreSQL container with logical WAL level and decide to use the built-in pgoutput plugin.
Logical decoding is what exposes committed changes to Debezium. The pgoutput plugin needs no extra installation in the database.
Explained in: Debezium connector for PostgreSQL :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: A query of the WAL level setting returns logical and the database accepts connections from the Connect container.
4. Decide how Debezium topics are created in your local Kafka: broker auto creation or explicit topics with chosen partitions and retention.
Topic settings decide ordering and how long history is kept. Both matter once the log is treated as the source of truth.
Explained in: Installing Debezium Kafka Connect Connectors :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You can state, for your stack, who creates the change topics and with what partition count and cleanup policy.
Steps
1. Run your first PostgreSQL connector against a small orders table that already holds a few rows, and watch the initial snapshot.
The snapshot shows how Debezium gets a consistent starting point before it follows the log. This is the first run every team does.
Explained in: Debezium connector for PostgreSQL :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: A topic for the orders table exists and holds one read event per existing row.
2. Run an insert, an update and a delete on the orders table and query the topic to read the three streamed change events.
This is log based change data capture in action: committed changes arrive in commit order without the application publishing anything.
Explained in: Debezium connector for PostgreSQL :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You see three new events in commit order, each with an operation type, before and after state, and a log position.
3. Compare the architecture of the CockroachDB connector with your running PostgreSQL connector, mapping source log, connector task, topic, event key, event value and offsets in both.
The same object model recurs in every Debezium connector. Seeing it against a second database separates the concepts from PostgreSQL details.
Explained in: Debezium connector for CockroachDB :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You have a two-column table that names the equivalent of each object for both connectors.
4. Run the MySQL connector against a local MySQL container with the same orders table and compare how it persists schema history with what the PostgreSQL connector does.
Binlog capture needs a persisted schema history topic to interpret old log entries, where logical decoding does not. The difference affects how you operate and recover each connector.
Explained in: Debezium connector for MySQL :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You can show the schema history topic for MySQL and explain why the PostgreSQL connector has no such topic.
5. Reproduce the dual write problem with a script that commits an order row and then publishes to Kafka as a separate write, kill it between the two, then write down the outbox message that should replace the second write.
Seeing the database and the topic disagree makes the failure concrete. The example message shows what must instead travel inside the transaction.
Explained in: Outbox Event Router :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You have one order in the database with no matching Kafka message, and a written outbox message with id, aggregate type, aggregate id, type and payload.
6. Create the outbox table with the expected columns and change the script to insert the order row and the outbox row in one local transaction.
The transactional outbox table makes the business change and the event atomic. A crash now loses both or neither.
Explained in: Outbox Event Router :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: Killing the script at any point never leaves an order without an outbox row or an outbox row without an order.
7. Configure the outbox event router transformation on a connector that captures only the outbox table, and run the script again.
The router turns raw outbox row changes into clean domain events, routed to a topic per aggregate type and keyed by aggregate id. This preserves per-aggregate ordering.
Explained in: Outbox Event Router :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: An order event topic receives messages whose key is the order id and whose value is only the payload.
8. Run the Debezium JDBC sink in upsert mode from the routed topic into a second database, then restart the Connect worker mid-stream to force redelivery.
Delivery is at least once, so a safe consumer must be idempotent. Upserts keyed on the primary key make replays harmless.
Explained in: Debezium connector for JDBC :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: After the restart the target table has exactly one row per order, with no duplicates and no errors in the sink status.
9. Enable signaling, drop the derived orders table in the second database and trigger an incremental snapshot of the source table to rebuild it while streaming continues.
Derived data is disposable when the log and the source remain the truth. Rebuilding on demand without stopping capture is how teams repair or add downstream views.
Explained in: Debezium connector for PostgreSQL :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: The derived table is repopulated to match the source row count while new live changes keep arriving.
10. Read how the Oracle connector mines from a downstream mining database; this needs a licensed database setup, so use your local PostgreSQL replication slot as the closest substitute and note the parallels.
Promoting capture off the primary onto a dedicated log reader is a common step before production. It shows how the capture path is moved without changing the events.
Explained in: Debezium Connector for Oracle :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: You can explain in three sentences where the log is read in that topology and what load it removes from the primary.
11. Package a pinned Connect image that includes the scripting libraries for content-based routing, keep connector configurations as files in version control, and recreate the whole stack from scratch with a routing rule applied.
A pinned image plus configuration in files makes the pipeline reproducible in CI and on any laptop. Content-based routing only works when its extra libraries are baked in.
Explained in: Content-based routing :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: A fresh checkout brings the stack up with one command and events matching your expression land in the alternate topic.
12. Configure a dedicated least-privilege replication role for the connector, switch the connector to it, and rerun the outbox flow end to end.
Production databases do not hand superuser credentials to a connector. Running with only replication and select privileges is the baseline for a production-like setup.
Explained in: Debezium connector for PostgreSQL :: Debezium Documentation (debezium.io), checked 2026-10-13
Done when: The connector status is running under the restricted role and a new order still produces exactly one routed event and one derived row.