You're back at BridgeCredit. Mei-Ling Tan signed off on last project's credit scoring fix and now wants the operating model that will run it -- and the six other models behind it -- for years: a model registry with real lifecycle states, an automated retraining pipeline that can be trusted, an A/B testing harness that can prove a candidate is actually better, and a rollback path that exists before the day you need it.
The discipline skills: designing a model registry as a lifecycle (staging, production, archived) where every transition is an evaluation decision with documented rationale, designing retraining trigger logic that filters monitoring signals instead of reacting to all of them, designing A/B tests with traffic splitting, segment stratification, and real significance testing, and configuring multiple MCP connections with deliberate permission scopes and trust evaluations.
The AI-direction lesson: when AI is reading and writing production state on your behalf, permission design decides whether the infrastructure is an asset or a liability. AI generates retraining pipelines that deploy-forward only -- no rollback, no eval gate against the current production model. It proposes auto-retrain on any monitoring signal without filtering input drift from real performance degradation. It declares A/B winners from point estimates without significance testing. It configures MCP connections write-capable when read-only would do. You drive all four.
Your Role
MLOps engineer and permission architect on a model already in production. Mei-Ling is treating this engagement as the template for everything BridgeCredit builds next, so the work has to be reusable, not just correct. You design the retraining pipeline, build the registry, run the A/B harness against live traffic, and document every transition so the next practitioner can read the system's history.
Scaffolding keeps the full-discovery shape from last project. The discipline shield is gone -- last time the operations side rested while you worked the data side; this time it reverses.
What's New
Last time you investigated a drifting credit scoring model, traced the drift to training-serving skew, refactored every feature into a shared module, extended the CI/CD eval gate, and stood up feature distribution monitoring.
This time you carry that infrastructure forward. The shared feature module, the parity gate, and the monitoring you built are what make this project safe to do at all -- without them, every retraining run would re-introduce the skew. What's new is the operations terrain on top: model registry lifecycle, retraining trigger design, A/B significance testing, and a multi-MCP permission architecture where each connection is a deliberate decision about scope and trust.
The hard part is judgment. AI implements every path you ask for competently. It does not tell you whether a 3% accuracy decline justifies a retraining run, whether an A/B difference is signal or noise, whether the registry MCP server you're about to install is trustworthy, or whether the candidate model's "win" came from real improvement or from seeing an easier slice of traffic. Mei-Ling and Marcus Webb -- the staff MLOps engineer who appears when trigger logic and significance design get hard -- both expect you to know.
Tools
- Claude Code -- primary AI agent, project memory travels with you (familiar)
- Python / pandas / scikit-learn -- existing model stack (familiar)
- MLflow -- experiment tracking (familiar); model registry features are new use
- MLflow MCP server -- active since earlier projects; permission and trust evaluation are new at this height
- Model registry MCP server -- new, with read-write permission design
- Monitoring MCP server -- new, with read-only permission design
- Retraining pipeline orchestrator -- a thin Python orchestrator over the existing training script and the registry (new)
- statsmodels / scipy.stats -- A/B significance testing (new use)
- FastAPI -- existing serving endpoint, extended with traffic-splitting middleware
- Docker, pytest, Git/GitHub -- carried forward; pytest extended with registry-state and A/B-decision-logic tests
- A second AI coding agent -- Codex CLI or equivalent, used in one unit only to make MCP's cross-platform behavior experiential
Materials
You receive:
- The fixed credit scoring model from last project -- in production with the shared feature module, parity gate, and distribution monitoring running
- A
models-scattered/directory of undocumented model files -- the registry mess Mei-Ling's on-call lead is complaining about - Six months of
inference-logs/for A/B simulation and monitoring backfill - The existing MLflow installation with experiment history
- An install script for the registry and monitoring MCP servers (you configure and connect; you do not write the server)
- Mei-Ling's forwarded email chain -- her terse cover note plus her on-call lead's longer note about the model files
- Your own CLAUDE.md and AGENTS.md carried forward, ready to extend with operations rules