You're building a hybrid data pipeline for Ashbridge Cement, a two-plant cement manufacturer in the US Midwest — combining daily ERP batch extracts with live kiln sensor streams so the production team can finally see quality results and kiln conditions in the same place.
The discipline skills: batch extraction with checkpoint-based resumption and object-storage landing zones, hybrid batch/CDC handoff design across an overlap window, Kafka consumers with manual offset commit, multi-consumer schema design with materialization chosen per consumer, cross-tool lineage from Dagster through dbt to BigQuery, and structured incident diagnosis.
The AI-direction lesson: you are the architect now. AI builds each piece competently — the batch extractor, the CDC consumer, the dbt models — but it does not design the seams where those pieces meet. Batch and CDC composed without a handoff design produce a gap or an overlap. One wide mart serves three consumer surfaces badly. Three agents building in parallel produce locally correct output that does not compose. Your job at this altitude is the integration AI cannot infer from either side alone — the handoff strategy, the multi-consumer schema contract, the interface between agents. You also design the infrastructure that holds the work together: path-scoped project memory, phase-scoped tool connectivity, agent-specific context, and hook selection across the full tier spectrum.
Your Role
You're Rachel Thornton's hybrid pipeline architect. She has two data velocities across two plants with two different streaming protocols, and she wants them unified without being told how. Her brief is short and direct — the problem, not the recipe. The architecture belongs to you.
The relationship with AI has shifted again. You supply the architecture; AI implements within it; you verify at the seams. Verification judgment — which technique fits which check — is now your active decision, not a checklist.
What's New
Last time you built a Debezium CDC pipeline with PII governance enforced across every surface, using composed multi-agent verification as a new mid-project concept.
This time the brief is thin and nothing is pre-committed. Batch versus CDC per source, which consumer surfaces to build, which materialization per model, which multi-agent orchestration pattern to use — all of that is yours to design before AI builds anything. Composed verification is no longer new; you deploy it by judgment. The infrastructure itself becomes architecture: different directories carry different rules, hooks span a spectrum from deterministic scripts to full-agent verification, and portability testing in a second AI coding agent is how you find out whether what you built survives outside the tool you built it in.
The hard part: hybrid pipelines fail at the seam. Batch and CDC can each be individually correct and the composed pipeline still wrong — a gap where the backfill ends, duplicates where CDC picks up, late-arriving events silently dropped into a closed partition, messages lost or duplicated during a consumer group rebalance. Three agents producing locally correct output can compose into a grain mismatch that passes every component test. Nothing throws an error. You have to design for it.
Tools
- PostgreSQL — the ERP source at Ashbridge (established)
- Apache Kafka (Docker Compose, KRaft mode) — streaming platform (established)
- Debezium — CDC connector, now used alongside batch (established from P11)
- MQTT broker (Eclipse Mosquitto) — new, bridges plant sensor streams into Kafka
- MinIO — local S3-compatible object storage as the landing zone (new architectural pattern)
- dbt Core — transformation layer (established)
- BigQuery — warehouse (established)
- Dagster — orchestration, now the primary surface for cross-tool lineage (established)
- GitHub Actions — CI/CD (established)
- Claude Code — primary AI coding agent, with path-scoped project memory, phase-scoped MCP connectivity, the full hook tier spectrum, and a multi-agent orchestration pattern
- A second AI coding agent (Codex CLI or Cursor) — for portability testing and for running the review agent in the multi-agent pattern
Materials
You'll receive:
- Project governance files — an AGENTS.md carrying the cross-cutting conventions, imported by a CLAUDE.md that adds the Claude Code briefing, and three path-scoped rule files in
.claude/rules/(one each for the staging, intermediate, and mart directories) that load only when a file under that directory is touched - PostgreSQL dump — Ashbridge's ERP production data (warehouses, products, batch records, quality tests)
- Kiln sensor samples — a sample of each plant's kiln feed, in the shape its control system exposes it
- Schema design template — an empty interface contract you complete before the multi-agent build begins
- MCP configuration — pre-built for the source database, ready to scope by workflow phase
- Hook scaffold directory — one example per hook tier (deterministic, external-service, fast-model, deep-agent), every one a
type: commandhook, to select from rather than use wholesale - Docker Compose stack — PostgreSQL, Kafka, Debezium Connect, MQTT broker, MinIO, Dagster
Rachel's first-contact email arrives in the chat. Everything else is your design.