You're building a CDC pipeline for a pharmaceutical distributor in Port Louis, Mauritius — capturing inventory changes from PostgreSQL through Debezium into Kafka, then into the transformation layer, with PII governance verified across every surface.
The discipline skills: Debezium CDC for PostgreSQL (WAL capture, before/after images, connector monitoring), idempotent MERGE keyed on primary key and LSN, automated PII scanning in CI via GitHub Actions, and audit trails for governance evidence.
The AI-direction lesson: composed verification. Multiple agents have contributed to this pipeline — and "each agent's output passes its own tests" does not mean "the composed pipeline works correctly." The verification challenge shifts from checking individual components to checking that pieces produced by different agents compose into a working whole. CDC adds a specific dimension: the WAL captures every committed change, including PII fields. Your mart-layer masking cannot reach Kafka topics or landing tables. A pipeline that works correctly and delivers real-time inventory data can simultaneously leak patient names through surfaces you haven't governed — with no error, no warning. Verifying governance at every surface, not just the output layer, is the culmination of everything you've built.
Your Role
You're building Vikram's CDC pipeline. This is full discovery — he mentioned CDC and compliance in a rushed voicemail, but the specifics come from asking. Nobody hands you a brief.
What's New
Last time you stood up Kafka from scratch, built producers and consumers, counted messages end-to-end, and created your first custom agent. The infrastructure was yours to build.
This time the events come from the database, not your code. Debezium reads the PostgreSQL write-ahead log and publishes every committed change as a CDC event. You do not control what gets captured — the database schema determines the event shape. That includes patient names, prescription data, and employee records that flow through Kafka topics where your mart-layer masking cannot reach.
The hard part: CDC captures everything. PII appears in surfaces you have not governed before. A pipeline that works correctly and delivers stale-free inventory data can simultaneously leak patient names through Kafka topics, landing tables, and generated documentation — with no error, no warning, and no one noticing until the regulatory inspector does.
Tools
- PostgreSQL — source database with CDC-enabled tables
- Debezium — CDC connector for PostgreSQL (new this project)
- Apache Kafka (Docker, KRaft mode) — streaming platform
- dbt Core — transformation layer
- BigQuery or DuckDB — warehouse
- Dagster — orchestration
- GitHub Actions — CI/CD for governance checks (new at this altitude)
- Claude Code — AI directing, MCP connections, hooks, agents
- Git / GitHub — version control
Materials
You'll receive:
- PostgreSQL database dump — the pharmacy inventory schema with seed data across 6 tables (inventory, prescriptions, controlled substances, temperature excursions, warehouses, pharmacies)
- Debezium connector template — a configuration template with placeholders for table scope, column filtering, and heartbeat intervals
- Project governance file — CLAUDE.md with the project context, PII inventory, work breakdown, and verification targets
Help available
A senior data engineer, Priya Venkatesh, is available as a chat resource throughout the project. She brings a pattern lens to CDC and PII handling. She's terse and won't direct your choices — ask her when you want a sanity check on grain, masking placement, or composed verification scope.