You're building a streaming data pipeline for a gold mine processing plant near Bartica, Guyana — ingesting sensor readings through Kafka, landing them in BigQuery, transforming them through dbt into anomaly detection, and alerting on equipment problems in minutes.
The discipline skills: Kafka fundamentals (topics, partitions, consumer groups, offsets, at-least-once delivery), Docker Compose for multi-container infrastructure, streaming-specific monitoring (offset lag, consumer group health), and end-to-end count verification between producer and consumer.
The AI-direction lesson: delegation. You'll design your first custom AI agent — not directing AI in real time, but defining a worker: what it does, what it can access, what it cannot touch. A data quality agent with read-only staging access reports findings for you to evaluate. The same agent with write access might "fix" issues it finds. The scope decision is yours. You'll also encounter a second AI tool for the first time, and discover that the infrastructure patterns you've built — project memory, skills, MCP connections — transfer across tools. The other hard part: AI defaults that work on first run don't survive a restart. Docker volumes, Kafka offset persistence, consumer commit behavior — the configuration AI generates passes a demo. You verify by restarting.
Your Role
You're building Nalini's streaming pipeline. Everything so far has been batch. Now the pipeline never stops running. Health isn't "did it complete?" but "is it keeping up?" You design a custom agent and encounter a second AI tool.
What's New
Last time you built enforceable data contracts with CI/CD enforcement, wrote your first skill and hooks, and extracted semi-structured data from MongoDB. The pipeline's promises became code.
This time the ingestion paradigm changes entirely. Kafka introduces continuous event streaming — messages flowing through topics, consumers tracking their position with offsets, and a delivery guarantee (at-least-once) that means duplicates are a system property, not a bug. Docker Compose stands up the multi-container infrastructure. The default configuration AI generates works on the first run. It may not survive a restart.
The hard part is that streaming failures are silent in a way batch failures aren't. A consumer that commits its offset before writing to the database loses messages without raising an error. You find out by counting — produced on one side, consumed on the other — and if the numbers don't match, nobody told you.
Tools
- Apache Kafka (Docker, KRaft mode) — streaming platform (new this project)
- Docker / Docker Compose — container infrastructure (new this project)
- Python — producer and consumer scripts
- BigQuery — cloud warehouse
- dbt Core with BigQuery adapter — transformation
- Dagster — orchestration and streaming health monitoring
- DuckDB — local development
- Claude Code — AI directing, custom agents (new: delegation)
- Codex CLI — second AI tool (new this project)
- Git / GitHub — version control
Materials
You'll receive:
- Historian export — 24 hours of sensor readings from the processing plant, with known anomalies embedded in the data
- Sensor metadata — 25 sensors with their types, equipment assignments, normal operating ranges, and alert thresholds
- Project governance file — CLAUDE.md with the project context, work breakdown, and verification targets