Reserve barrels sell for four times the price of standard, and the winemaker at a family winery in Mendoza, Argentina tastes hundreds of samples by hand to find them. You build a classification model that predicts which samples are likely to clear the Reserve bar, so she can spend her tasting time on the borderline cases.
The discipline skills: building logistic regression and decision tree classifiers with scikit-learn, evaluating with precision, recall, confusion matrices, and ROC curves, tuning a classification threshold to match the client's cost priorities, and catching proxy features that give the model the answer.
The AI-direction lesson: AI defaults to accuracy as the evaluation metric. On this data, where only about 8% of barrels make Reserve, a model that never predicts Reserve achieves 92% accuracy and catches nothing. The skill is recognizing when AI's default metric is meaningless -- and this is the clearest example you have encountered so far. The number looks right. The metric is wrong. Once you see this pattern, you start checking AI's metric choice before interpreting its results.
Your Role
You evaluate the model honestly -- not with a single accuracy number, but with metrics that show exactly what the model catches, what it misses, and what it falsely flags. The scaffolding is the same as last time. What changes is the terrain: classification on imbalanced data requires different evaluation from everything you have done before.
What's New
Last time, you combined three datasets, ran inferential analysis with assumption checking, and built a prediction model with regularization. You learned to verify through cross-model review and to communicate with effect sizes.
This time, the data is a single source — no joining work. But only about 8% of barrels make Reserve. That imbalance changes everything about how you evaluate a model. A model that never predicts Reserve achieves 92% accuracy and catches nothing. The metrics that worked for regression do not apply here. Precision, recall, confusion matrices, and ROC curves replace RMSE and R-squared.
The hard part is not building the model. It is figuring out whether the model actually does what Luciana needs — and that question depends on which errors she can live with.
Tools
- Python 3.11+ via your conda "ds" environment
- Jupyter Notebook for the analysis
- pandas for data handling
- scikit-learn for classification models, confusion matrices, ROC curves, and precision/recall
- scipy for statistical checks
- matplotlib / seaborn for visualization
- Claude Code as the AI you direct
- Git / GitHub for version control
Materials
You receive:
- A barrel production dataset with five years of production data and quality scores
- A data dictionary explaining each column
- A methodology memo template to fill in as you work
- A project governance file (CLAUDE.md) for Claude Code
- Luciana's voicemail explaining what she needs
Same scaffolding as last time: templates, verification guidance, cross-model review. No provided answer key.