You're building an analytical pipeline for a textile dyeing operation in San Salvador — transforming raw production batch records into quality metrics that reveal which variables drive color match scores across three production lines.
The discipline skills: window functions for tracking operator trends over time, joins between tables at different grains, reusable Jinja macros in dbt, and layered quality monitoring with Soda Core alongside dbt tests.
The AI-direction lesson: planning before prompting. This is the first project where you decompose the work before starting, using plan mode to sequence the steps. You'll also create the project's CLAUDE.md — the data dictionary and naming conventions that shape every model AI produces. The difference between directing AI with that governance file and directing AI without it is the difference between specific, testable constraints and vague compliance. A prompt that says "follow dbt best practices" produces different output from one that says "all staging models: stg_ prefix, source-conform only, no joins, no business logic." The infrastructure you build before the pipeline determines the pipeline's quality.
Your Role
You're building Roberto's quality analysis pipeline. The transformations are more complex than previous projects — window functions, multi-grain joins, Jinja macros. Soda Core enters for trend analysis. You'll plan the work before starting, and create the project governance file yourself.
What's New
Last time you connected three data sources for Rashida's orthopedic-device UDI chain-of-custody pipeline, resolved identity across systems, and orchestrated with Dagster for the first time. The pipeline was complex because of multiple sources and identity resolution — but the transformations themselves were straightforward joins and aggregations.
This time the sources are simpler (one CSV), but the transformations are harder. Window functions look correct row-by-row but can produce wrong results in aggregate. Joining tables at different grains silently inflates totals. You'll encounter numbers that look right and aren't — and the only way to catch them is to run the pipeline twice and compare, or to check against Roberto's domain knowledge.
Soda Core enters as a new tool for quality monitoring. dbt tests tell you whether individual values are valid. Soda Core tells you whether today's batch looks normal compared to yesterday's. They catch different kinds of failures.
Data governance enters for the first time. Roberto's data includes operator-level performance metrics. The question of who should see that data — and whether the operators would want their individual performance visible on a dashboard — is worth considering.
Tools
- dbt Core + dbt-duckdb adapter — transformation framework
- DuckDB — local analytical database
- Soda Core — quality monitoring with trend analysis (new this project)
- Dagster — orchestration with freshness policies and materialisation scheduling
- GitHub Actions — CI/CD quality gates (new this project)
- Claude Code — your AI agent, with plan mode for decomposition
- Git / GitHub — version control
Materials
You'll receive:
- Batch data — a 30-row sample for initial exploration and a ~900-row full dataset spanning 30 days of production across all three lines
- Pipeline spec template — requirements pre-filled from Roberto's brief, design sections empty for you to complete
- CLAUDE.md template — the project governance file structure, yours to fill with the data dictionary and conventions you establish
- Soda Core configuration guide — installation, check file structure, running scans
- GitHub Actions template — CI/CD workflow for dbt tests and Soda Core checks