Diego Rivas runs a poultry processing cooperative in northern Nicaragua — five plants on five different systems, no warehouse chosen, no orchestrator chosen. He needs one number for total cooperative production, an export-certification trail every kilo can be traced through, and a fair plant-by-plant comparison.
The discipline skills: designing a data quality architecture as its own deliverable — what to test, at what grain, with what tools, at what layer, at what cost — and running the same dbt logic on BigQuery and Snowflake, the same daily refresh on Dagster and Airflow, long enough to see where each pair disagrees. Per-engine and per-orchestrator cost capture on a calibration window. An export-certification audit trail that survives whichever engine wins.
The AI-direction lesson: "the same logic on a different engine" does not produce the same answers. AI will write correct BigQuery SQL and correct Snowflake SQL and assert the totals will match. They will not — dialect-level differences in numeric precision, null-in-aggregate semantics, and string equality and collation can silently diverge across warehouses, and which of those classes actually bites is what the bake-off has to find out. Trust calibration becomes per-dialect. The quality architecture is the layer that has to hold while the platform underneath is still being chosen — so you design it first.
Your Role
You are the consolidation engineer evaluating two warehouses and two orchestrators head-to-head before recommending one. The cooperative cannot specify a stack; the plants will not switch their own systems. The work is comparative, not greenfield.
Scaffolding holds at the level it has been since P11 — you bring your own dbt patterns, Dagster setup, MCP config, hooks, and path-scoped governance file from thirteen prior projects. There is no inherited architecture and no exemplar. You direct AI through dialect-specific and paradigm-specific work: per-engine agents, per-orchestrator agents, verification agents that diff outputs across them. You decide where running both stacks in parallel earns its cost.
What's New
Last project you shipped the backfill on the Ashbridge Cement platform — PySpark for the historical volume, Delta Lake on the landing zone, a cost-bounded execution plan, and the cross-engine verification technique that found rounding drift between dbt and PySpark.
This project the cross-engine technique returns as something you already know how to use, applied to a different axis: warehouse versus warehouse. Genuinely new is the alternative warehouse (Snowflake), the alternative orchestrator (Apache Airflow), the quality architecture as a design problem in its own right, and the cooperative shape itself — five plants, five managers, one client. JT Thompson appears as senior colleague — terse, infrastructure-savvy, has switched orchestrators in production before and pushes you to express trade-offs in cost terms, not aesthetic ones.
The hard part: cross-engine drift is silent. The same SQL on BigQuery and Snowflake returns different totals with no error. The same daily refresh on Dagster and Airflow looks fine on both UIs and answers different operational questions. Quality checks tuned to one engine fire false positives on the other. Running both stacks on the production cadence is a budget hole. You design for all of it on the calibration window.
Tools
- dbt, BigQuery, Dagster, Soda Core, DuckDB, MinIO, Docker Compose, GitHub Actions, Claude Code, MCP, Codex CLI — all established from prior projects
- Snowflake — alternative warehouse, through dbt's Snowflake adapter on a free-trial account for the calibration window
- Apache Airflow — alternative orchestrator, run locally via a Docker Compose overlay
- A plant-source generator — produces five realistic plant exports across heterogeneous systems (cloud database, MySQL, Excel, Access-style CSV)
- A quality observability dashboard stub — renders Soda Core results, dbt test results, and freshness signals in a single view
Materials
You'll receive:
- A cooperative governance file — extends your CLAUDE.md pattern with
snowflake/andairflow/path-scoped blocks - Five-plant source generator — produces Plant 1 cloud-DB extracts, Plants 2-3 MySQL dump, Plant 4 Excel sheet, and Plant 5 Access-style CSV with ambiguous-date and collation pitfalls preserved
- Snowflake setup notes — free-trial account through dbt's adapter, scoped to the calibration window
- Airflow Compose overlay — local single-scheduler, single-worker stack that reads your existing dbt project
- Per-engine agent context templates — one Snowflake-aware, one Airflow-aware
- Quality architecture template — inventory shape (what / grain / tool / layer / cost) plus engine-portability and orchestrator-portability flags
- Comparison spec template — the cross-engine and cross-orchestrator comparison structure
- Quality observability dashboard stub — a directory you configure against the project's local stack
- Carolina's forwarded email — the cooperative's administrative office hands you off to Diego, whose introduction follows
The schema design document, hook scaffolds, MCP config, AGENTS.md, base Docker Compose, and the path-scoped governance pattern all carry forward from your existing repository. You extend, you do not recreate.