A commercial airline in the UAE asks for a 24-hours-before-departure delay probability model -- but the deliverable is not a single "best" model. The deliverable is a ranked flight list with a per-flight reliability signal that tells Operations which predictions they can act on and which they should treat as unknown.
The discipline skills: pipeline-disciplined preprocessing with scikit-learn, imbalanced binary classification evaluated at operationally relevant precision-recall targets, a principled specification set that varies substantive preparation choices one at a time, MLflow experiment tracking, and composition verification across the specifications.
The AI-direction lesson: AI's default in prediction work is to produce one preparation pipeline, fit one model, report the best-looking metric, and call that the answer. In this project that default is the professional failure. The work is to refuse the single answer -- design the specifications yourself, hold the methodological guardrails constant while varying the substantive choices, track each run so the comparison is reproducible, and make the spread across specifications the headline finding. Preprocessing leakage (normalisation and imputation fit on the full dataset before splitting) is AI's subtlest prediction error and you'll catch it here. Robustness and fragility become the primary vocabulary of the deliverable.
Your Role
You deliver a family of models, not a model. The per-flight output carries both a predicted probability and a reliability signal derived from how the prediction moves across specifications. Analytically familiar territory -- cross-sectional prediction -- now carries a new frame: the verification IS the sensitivity analysis, and the design of the specification set is the methodological decision the work is built around.
What's New
Last time, you forecast monthly container volumes for Carlos at the port authority in Lisbon. Temporal discipline was the load-bearing idea; you wrote your first skill and configured your first hooks to make that discipline automatic.
This time, the terrain flips. The analytical methods (logistic regression, gradient boosting, cross-validation) are familiar. What is new is multi-specification thinking: designing four or five substantive variations, running each as a tracked experiment, verifying they compose correctly, and reporting the spread as the finding. Preprocessing leakage enters as a distinct concept from the temporal leakage you've met before -- subtler, more common, and the one AI will produce by default.
The hard part is the judgment about what varies and what holds constant. Trivial variation -- different random seeds, cosmetic hyperparameter tweaks -- is not sensitivity analysis. Substantive variation -- which features, which imputation, which threshold -- is. AI will not draw that line for you.
Tools
- Python 3.11+ via your conda "ds" environment
- Jupyter Notebook for the analysis
- pandas for data handling
- scikit-learn -- familiar library, with focused use of
PipelineandColumnTransformerfor leakage-free preprocessing - MLflow (new) -- experiment tracking for the specification family; the unit that introduces it walks through the setup
- statsmodels (carry-forward, lighter use)
- matplotlib / seaborn for visualisation
- DuckDB and DuckDB MCP server (carry-forward, available if the operations data is large enough to benefit)
- Claude Code -- the assumption-checking skill and pre-evaluation hook from last project carry forward and stay active
- Git / GitHub for version control
Materials
You receive:
- Four calendar years of flight operations data (~280K flights across 120 routes)
- Daily weather data for the top 40 airports, with realistic gaps
- Route metadata including first-operated dates (15 routes have less than 12 months of history)
- A data dictionary
- A project CLAUDE.md with the analytical conventions for this engagement
- A stub MLflow configuration pointing at a local tracking store
No methodology template, no pre-designed specification set, no pre-selected threshold.