Rachel Thornton is back. The hybrid pipeline you built for Ashbridge Cement is running and her quality team has found patterns worth investigating, but they need history — three years of 30-second kiln readings across 12 sensors at two plants, plus four years of ERP production batches. The current pipeline only covers from the day it went live.
The discipline skills: choosing where PySpark belongs and where dbt is still the right engine, designing a backfill that does not duplicate or block the live pipeline, running the work inside a cost ceiling Rachel can defend to her CFO, and verifying that two transformation engines produce the same numbers on the windows where they overlap.
The AI-direction lesson: backfill is the stress test, not a feature. Every design decision the previous project made is now load-bearing under volume and a deadline. AI will write competent PySpark and competent dbt and assert that "the same logic" produces the same numbers — until you diff the output rows and find the rounding difference, the null-handling difference, the MERGE semantic that quietly shifts a sum. Trust calibration becomes per-engine. Cost becomes a gate, not a metric.
Your Role
You are the operating engineer for the platform you already built. The architecture is not up for redesign — Rachel's brief is short because she is asking for a delta against last project. You extend the path-scoped governance file with a pyspark/ block, add the backfill plan and the cross-engine verification spec as new persistent context, and run the historical reprocessing under a budget Rachel signs off on before the first job kicks off.
The relationship with AI extends across engines. You already direct a build agent and a review agent; now you add a PySpark agent and a cross-engine verification agent. Each gets context scoped to its job. You verify at the seams.
What's New
Last project you shipped the hybrid batch and CDC pipeline — extraction, the batch-to-CDC handoff, three consumer marts on BigQuery, cross-tool lineage in Dagster, and a multi-agent build pattern with a schema design document as the interface contract.
This project the architecture is the inherited working system. What is new is operational and technical: a second transformation engine for volume that no longer fits dbt-on-warehouse, a transactional storage layer on the landing zone, a backfill plan that lives inside a cost ceiling, and a verification class — cross-engine output reconciliation — that did not exist before. Priya Venkatesh, the senior data engineer you have met before, is briefly available. She warns about cost early and pushes for cross-engine verification before any backfilled data reaches the consumer marts.
The hard part: backfills fail in ways that do not throw errors. Two engines run "the same logic" and produce different sums. The cutover overlaps the live pipeline and an event lands twice — or not at all. PySpark code that handles a sample correctly can OOM on the production partition if a partition key carries disproportionate data — the engineering instinct is to anticipate skew when designing partitions, even when this fixture's plants happen to contribute comparable row counts. Nothing alerts. You design for it.
Tools
- PostgreSQL, Kafka, Debezium, MQTT, MinIO, dbt Core, BigQuery, Dagster, GitHub Actions — all established from last project
- PySpark — second transformation engine for high-volume work
- Delta Lake — transactional storage layer on MinIO (supports MERGE, time travel, schema evolution)
- Local Spark cluster in Docker Compose — single-node master plus one worker, standing in for a production cluster
- Claude Code — primary AI coding agent, with a new
pyspark/block in the path-scoped project memory and a per-engine agent context file - A second AI coding agent (Codex CLI or Cursor) — the portability check from last project, re-run once at close
Materials
You'll receive:
- Updated project governance file — extends your existing CLAUDE.md with a
pyspark/path-scoped block, new verification targets, and the dual-engine commit convention - Historical data generator — a Python script you run once to seed three years of historian-style kiln readings (~38M rows per plant, ~76M total) and four years of ERP production batches into the local stack
- Spark and Delta Lake Docker overlay — adds a single-node Spark master plus one worker to your existing stack, with Delta Lake configured against MinIO
- Per-engine agent context files — one for the PySpark agent, one for the cross-engine verification agent
- Templates you fill in this project — backfill context, backfill plan, cross-engine verification spec, cost budget worksheet, cutover checklist, and a dual-engine README scaffold
The schema design document, the hook scaffolds, the MCP config, the AGENTS.md, the base Docker Compose file — all carry forward from your existing repository. You extend, you do not recreate.
Rachel's follow-up email arrives in the chat.