You're building a pipeline for a grain storage company in Kazakhstan's Kostanay region — combining CSV storage data from two elevators with weather data extracted from the Open-Meteo API, designing a schema that brings both sources together in DuckDB, and verifying the combined output.
The discipline skills: extracting data from a REST API with error handling and silent-failure verification, designing a schema from source profiles (including the grain decision — what one row represents), and verifying row counts against the expected day count derived from the request range rather than a file count.
The AI-direction lesson shifts from checking output to decomposing work. Last project, you followed a provided sequence. This time, what to ask AI and in what order is your decision. A single request to "build the complete pipeline" produces worse results than focused, sequential requests — profile first, then design, then extract, then transform. And across those turns, AI forgets. It contradicts earlier decisions, re-introduces column names it was told to change. Noticing when context has degraded is a skill you start building here.
Your Role
You're building the pipeline that combines Assel's storage data with weather data from an API. You design the schema this time — profiling both sources, deciding the grain, and determining how they join. The pipeline spec tells you what Assel needs. How you decompose the work into tasks for AI is up to you.
What's New
Last time you loaded CSV files and verified row counts against a known target. This time, one of your sources is a live API. It returns timestamps in a different timezone, can silently swallow HTTP errors mid-extraction, and can deliver fewer records than you expect without raising. The "did everything arrive?" question is the same one you asked with Carlos's honey data. The verification technique is different.
The other new piece is schema design. You profiled nothing last time — the schema was documented and you implemented it. Now you profile both data sources, decide on the grain (what one row represents), and design the staging and mart layers yourself. If the grain is wrong, every downstream number is wrong.
The hard part is the extraction boundary. The API call completes, reports success, and you have data. Whether you have all the data is a separate question — and the API does not answer it for you.
Tools
- Python — via your Miniconda
deenvironment - DuckDB — analytical database for both storage and weather data
- SQL — for staging and mart transformations
requests— Python library for API extraction (new this project)- Claude Code — your AI agent, doing the implementation work
- Git / GitHub — version control
Materials
You'll receive:
- Pipeline specification — what to build, what Assel needs, verification targets
- Storage data — 6 months of CSV exports from both elevators
- Verification checklist — row counts, staging counts, spoilage-weather correlation spot-checks
- Project governance file — CLAUDE.md with project context, key references, and commit conventions