You're building a data pipeline for a regional water utility in Bendigo, Victoria — loading CSV sampling records from six catchment monitoring points into DuckDB, cleaning and deduplicating them in a staging layer, and transforming them into a mart table the utility can query for compliance reporting to EPA Victoria.
The discipline skills: loading CSV data into DuckDB with correct encoding and type coercion, writing SQL transformations for cleaning and deduplication, verifying row counts between source and target, and reading a schema specification before building anything.
This is your first time directing AI to build a data pipeline. AI will produce something that runs without errors — and that can still produce wrong numbers. A pipeline that completes successfully is not the same as a pipeline that's correct. The skill you're building here is checking AI's output against known targets: row counts, breach totals, specific values. AI won't tell you when the numbers are wrong. You find out by looking.
Your Role
You're building Sarah's data pipeline. Everything you need is provided — the pipeline spec, the schema, the expected output targets. Read them before you direct anything. Your job is to give AI the right context, review what it produces, and verify the numbers.
What's New
This is your first real pipeline. The work follows a pattern you'll use on every project after this: read the spec, direct AI, check the result against known targets.
The hard part isn't getting the pipeline to run. Claude Code will produce something that executes without errors. The hard part is knowing whether the output is actually correct. A pipeline that completes successfully can still produce wrong numbers — and it won't tell you.
Tools
- Python — via your Miniconda
deenvironment - DuckDB — lightweight database for loading and querying the monitoring data
- SQL — for transformations inside DuckDB
- Claude Code — your AI agent, doing the implementation work
- Git / GitHub — version control from the start
- VS Code — your editor
Materials
You'll receive:
- Pipeline specification — what to build and what "correct" means
- Schema documentation — the staging and mart layer structure, naming conventions
- Verification checklist — specific values to check your output against
- Sample data — CSV files from Sarah's monitoring points