You're auditing three production ML systems you did not build. James Holloway, the new CTO of Kerbside -- a Manchester food-delivery platform running about 9,000 deliveries a day -- inherited delivery-time estimation, restaurant ranking, and courier assignment, all degrading, with almost no documentation and no one left to ask. He wants an honest audit: what's broken, why, what to fix first, and the documentation so the next person doesn't start where he did.
The discipline skills: profiling data and tracing pipelines to find where a model's training and serving inputs diverge; recomputing honest, disaggregated metrics on a model you didn't train, against a baseline you reconstruct because none was written down; auditing a monitoring configuration to test whether a green dashboard is telling the truth; reconstructing an undocumented serving architecture and model lifecycle well enough to operate it; deciding, system by system, whether the fix is retrain, repair, or replace under a "can't fix everything at once" constraint; and writing the operator documentation that lets the next engineer run all of this in their first week.
The AI-direction lesson: everything you spent the track learning to build correctly, you now have to detect the absence of in someone else's work -- and the detection is harder than the building. AI is good at reading and profiling code at volume, which is exactly why it's dangerous here. Asked what an inherited pipeline does, AI produces a fluent, plausible, partly-wrong account. Asked to fix a bad prediction, it proposes the obvious fix without finding the root cause. It reads a monitoring config and reports "monitoring is in place" without asking whether the thresholds mean anything. It will fix one thing and quietly break another, because it holds no model of how the pieces relate. Your posture is investigation with distrust: every claim AI makes about the inherited system gets verified against evidence you produce, not accepted because it sounds right.
Your Role
System auditor and inheriting ML engineer. You diagnose what's wrong with evidence, decide what gets fixed and in what order, fix the priorities, rebuild the monitoring, and hand over documentation that survives your absence.
This is the thinnest scaffolding in the track. There is no brief, no design, no templates -- three running systems and a late-night text thread of symptoms. The client knows what's broken; he does not know why, and he can't tell you. The causes come from the audit, not from him. One piece of scaffolding returns: a senior colleague, Dr. Sarah Chen, on Slack. She's here for one reason -- auditing a system you didn't build is the one situation where "how would you approach this?" is a real methodology question, not a shortcut. She gives method, never the diagnosis.
What's New
Last time you designed and proved AI development infrastructure for Aroha's analytics consultancy in Auckland -- the infrastructure was the deliverable, and you built it from a sparse Slack thread with no senior colleague. The full directing repertoire carries forward, and so does the habit from P23 of proving a claim with evidence instead of asserting it.
The terrain inverts completely. In every project before this one you built something, even when the thing you built was infrastructure. Here you build nothing from scratch. Three ML systems already exist, in production, degrading, and you inherit them cold. The work is no longer construction -- it's diagnosis, triage, and handoff.
The hard part is that the indicators lie. The dashboards are green and the systems are failing, and AI's most confident explanations will be wrong exactly where it matters most. Finding the truth means working from the data forward, with discipline, against a system that looks fine on every screen James can see.
Tools
- Claude Code -- directing, used for investigation rather than construction
- Python -- the substance of the inherited systems you read, profile, and patch (pandas/scikit-learn feature and training code, a serving layer, monitoring config)
- Git/GitHub -- the inherited repositories, with sparse history; you commit diagnoses, fixes, and documentation
- Jupyter -- the audit notebook, where you profile data, recompute honest metrics, and produce the evidence
- A model registry (MLflow or equivalent) -- inherited and undocumented; you read it to reconstruct lifecycle state, you don't deploy it
- Your accumulated skills directory -- leakage-check, disaggregated-metrics, eval-gate skills, now turned on someone else's code
No new tool is introduced. Every tool is an investigation surface.
Materials
You receive:
first-contact-text.md-- James's symptoms-only text thread, sent late at night- Three inherited Git repositories -- delivery-time estimation, restaurant ranking, courier assignment -- each with real but undocumented Python and sparse commit history
CLAUDE.md-- the audit's working memory, mostly stubs that name what the audit must fill in- The inherited monitoring configuration -- stale, and not to be trusted
- An inherited model registry with undocumented promotion history
- Before/after fixtures for a third-party restaurant API schema change
- Data samples that exhibit the problems you'll need to find
audit-notebook.ipynb-- where the profiling and honest-metric work happens- Your own accumulated
skills/directory from prior projects
The audit method in practice, every diagnosis and its evidence, the triage and remediation plan, the fixes, the rebuilt monitoring, and the operator documentation are all yours to produce.