You're building CI/CD pipelines with evaluation gates and a drift detection system for the nurse-to-hospital matching model -- automated infrastructure that blocks bad models from deploying and catches when production data shifts away from what the model was trained on.
The discipline skills: configuring GitHub Actions workflows with evaluation gates that block deployment on threshold failure, implementing drift detection with Evidently AI (data drift vs. concept drift), calibrating alert thresholds for the specific use case, designing a response plan that connects alerts to operational decisions, and using branch-based workflows for model experimentation.
The AI-direction lesson: planning decomposition before directing becomes essential. CI/CD, drift detection, and response plans have dependencies -- the evaluation suite must exist before the gate can check it, the baseline must be computed before drift can be measured. Using Claude's plan mode to map this dependency chain before building anything produces better results than improvising. AI generates GitHub Actions workflows that run the evaluation suite but don't gate on the results -- the eval step succeeds regardless of whether the model passes. It sets drift thresholds at textbook defaults without calibrating to the use case. It generates alerts without response plans. The skill is planning the whole system before asking AI to build any single piece.
Your Role
Templates provide structure for the GitHub Actions workflow and the drift detection configuration. Guides are gone -- you fill the templates with your own judgment about which metrics to gate on, which features to monitor, what thresholds to set, and what the team should do when an alert fires.
What's New
Last time you built the matching model itself — Pipelines, transfer learning, a fairness audit. The model was the challenge.
This time the model is the starting point. The challenge is making it reliable in production. CI/CD with GitHub Actions automates evaluation: the pipeline runs the eval suite on every push and blocks deployment when the model does not pass. Drift detection with Evidently AI monitors whether the production data still looks like the training data. And the response plan connects alerts to decisions — what to do when something changes, not just that something changed.
The hard part is not the configuration. It is designing the system: which metrics should block deployment, which features matter enough to monitor, how sensitive the alerts should be, and what the team does when they fire.
Tools
- Python — scripting, drift analysis
- GitHub Actions — CI/CD automation (new)
- Evidently AI — drift detection (new)
- scikit-learn — evaluation suite (familiar)
- MLflow — experiment tracking (familiar)
- Git / GitHub — branch-based workflows (deepening)
- Claude Code — AI direction, plan mode (new use)
Materials
You receive:
- Production placement data with recent records from the changed hospitals
- Training baseline data for drift comparison
- A GitHub Actions workflow template with placeholder steps for evaluation and gating
- An evaluation suite script with configurable thresholds
- A drift detection configuration template for Evidently AI
- A response plan template with severity levels and response procedures
- A project governance file (CLAUDE.md)