The third-party embedding API behind a digital news platform's entire content-recommendation system is being deprecated in 90 days. Maria Santos, VP of Engineering at Balita Digital in Manila, needs you to evaluate replacements, migrate everything without downtime, prove recommendations do not get worse, and -- the part she hasn't formalized yet -- make sure the next deprecation isn't another fire.
The discipline skills: running discovery well enough to reframe what the client actually needs; designing an equivalence evaluation that measures recommendation quality on real reader engagement -- clicks, reads, dwell -- disaggregated by language, not on embedding cosine similarity; evaluating an unfamiliar embedding model against everything you know about durable interfaces; planning a zero-downtime cutover for three million cached vectors with a designed failure path; architecting monitoring that watches quality, cost, latency, and language coverage together; and building a provider abstraction so the next swap is a config change, not a rebuild.
The loop lesson: this is the project where you find out whether you learned patterns or products. AI is good at the volume work here -- re-embedding, scripting the backfill, scaffolding the harness -- which is exactly why its defaults are the trap. Asked to migrate, AI treats it as a one-line API change, proves equivalence with cosine similarity instead of asking whether readers still click, evaluates the new model in English and calls it done, and hardcodes the new provider so the next deprecation is another from-scratch fire. None of these are obviously wrong; all of them quietly burn the 90 days and the ad revenue. You supply the architecture; AI supplies the labor, against it, not its own.
Your Role
Migration and adaptation engineer. You discover the real requirement, design the proof that recommendation quality survives a model swap, choose a replacement on evidence, cut over without downtime, build the monitoring, and hand over infrastructure that makes the next deprecation a non-event.
This is the thinnest scaffolding the track has offered. No brief, no spec, no design pack, no templates. A voicemail, a constraint, and a deadline. There is no one to ask about the unfamiliar tool -- not because anyone was forgotten, but because adapting to a tool no one has used before is precisely the situation where there is no one to ask. The one safety line is narrow and deliberately stays out of the tool: a senior evaluation consultant reachable on-demand only for whether your "no degradation" proof is honest, per language -- she won't volunteer and won't help with the model itself. The pattern vocabulary you've built across the track is the methodology, and you are the only one carrying it. You direct AI to integrate and migrate against an architecture only you can supply.
What's New
Last time you inherited and audited three degrading production ML systems for James at Kerbside in Manchester -- diagnosing training-serving skew, a silent schema failure, and a lying dashboard, then triaging and handing over. The full directing-and-verification repertoire carries forward: evidence-first reasoning, discovery against a thin first contact, designing verification coverage before accepting any change, and documentation written for someone who didn't build the thing.
The terrain inverts. At Kerbside nothing worked and you diagnosed why. Here everything works -- and will stop working on a date. The forcing function is external, not an internal failure. There is nothing to debug; there is something to adapt before a clock runs out.
The hard part is that the obvious migration is the wrong one. "Swap the API and re-embed" is plausible, it's what AI proposes, and it ignores the language coverage, the cutover, the cost interaction, and the durability that's the actual point. Proving quality is preserved -- on the signal the business actually feels -- is harder than performing the swap, and the swap is the small part.
Tools
- Claude Code -- directing the evaluation, migration, and durability design
- Python -- the substance: the existing recommendation and serving code you read and adapt, the candidate model integrations, the re-embedding and backfill scripts, the evaluation harness
- Git/GitHub -- the project repository; you commit the harness, the provider abstraction, the runbook, the monitoring, and the durability documentation
- Jupyter -- the evaluation notebook, where you measure recommendation quality on reader-engagement data, disaggregated by language
No new tool category is introduced. The unfamiliar replacement embedding model is itself the adaptation surface -- which is exactly the point.
Materials
You receive:
first-contact-voicemail.md-- Maria's voicemail, left from her car in Manila trafficcurrent-architecture.mdanddeprecation-notice.md-- the system as it stands and the announcement that started the clockrecommendation-system/-- the real Python codebase built on the deprecating provider: the embedding step, the similarity and recommendation logic, the serving layer, the cached-embedding store and its contractembedding_candidates/-- offline, deterministic integration stubs for the deprecating model plus at least two replacement candidates, one of which you've never used; each stub generates reproducible outputs on demand without live paid APIs- A representative reader-engagement dataset -- clicks, reads, dwell, tagged by language -- the ground truth for whether quality degraded, plus a representative sample of the cached three-million-article embedding store with its contract
CLAUDE.md-- the migration's working memory, with the model-selection and migration sections as stubs you fill from the work
The discovery and reframe, the evaluation criteria and harness, the model selection with evidence, the zero-downtime plan and its failure path, the provider abstraction, the monitoring, and the durability runbook are all yours to produce.