You're building enforceable data contracts for a seed distribution company in Harare, Zimbabwe — one definition of every key metric for five consumer teams, freshness guarantees calibrated to actual needs, and schema specifications that CI/CD enforces on every pull request.
The discipline skills: writing data contracts as YAML specifications with schema, freshness, and quality guarantees, dbt model versioning for safe schema evolution, semi-structured data extraction from MongoDB, retention policies grounded in real regulatory requirements, and CI/CD enforcement that breaks the build on contract violations.
The AI-direction lesson: encoding constraints as deterministic infrastructure. You'll write your first skill — a reusable dbt development workflow that AI follows without you re-specifying each step. And your first hooks — a pre-commit hook that runs dbt tests, a pre-push hook that scans for PII. The difference matters: project memory tells AI what to do and hopes it complies. A skill makes it part of a workflow that fires when invoked. A hook enforces it automatically at a lifecycle point regardless of whether anyone remembers. Choosing which mechanism to use — memory, skill, or hook — is a design decision about how much you trust compliance versus enforcement.
Your Role
You're building Tendai's data contracts. This is the first project where you're given only a brief — no templates, no spec, no guides. You create everything: the pipeline specification, the contracts, the CI/CD enforcement, the ingestion design, the retention policies, and the AI directing infrastructure. The materials still ship with a CLAUDE.md governance file, but that's project memory for AI — client context, tech stack, verification targets — not an answer guide for the work.
What's New
Last time you moved to BigQuery and discovered that every query has a dollar cost. You designed partitioning and clustering as cost architecture, connected MCP to databases and pipelines, and built RBAC with column-level security.
This time the pipeline's promises become enforceable. Data contracts turn documentation into code — YAML specifications that CI validates on every PR. Freshness monitoring adds a quality dimension that row-level tests can't verify: data can be correct in every field and still be too old. You'll ingest semi-structured data from MongoDB for the first time, design retention policies grounded in real regulatory requirements, and evaluate build vs buy for managed ingestion.
The hard part is that contracts only matter when they're enforced — and enforcement has cost. Every CI run validates contracts. Every freshness check queries the warehouse. You'll calibrate between contract rigor and development velocity, and between regulatory compliance and operational simplicity.
Tools
- dbt Core with BigQuery adapter — data contracts and model versioning (new this project)
- BigQuery — cloud warehouse
- DuckDB — local development
- Soda Core — quality monitoring, freshness checks
- Dagster — orchestration
- MongoDB — semi-structured data source (new this project)
- GitHub Actions — CI/CD contract enforcement (new this project)
- Claude Code — AI directing with MCP, plus skills and hooks (new this project)
- Git / GitHub — version control
Materials
You'll receive:
- ERP export — delivery and order data with SAP material codes and legacy codes from the pre-migration era
- Field trials data — nested JSON from MongoDB with measurements, GPS coordinates, and farmer confidentiality conditions
- Logistics export — shipment tracking across four countries, including Canadian partner references
- Product code mapping — reconciliation table between SAP and legacy codes, deliberately incomplete for discontinued varieties
- Project governance file — CLAUDE.md with the project context and data source overview