Classical ML Production Architecture

You're designing the production architecture for a wafer defect-prediction system for Shu-Fen Lin, Director of Quality Engineering at PrecisionTek Components, a semiconductor manufacturer in Hsinchu, Taiwan. Her three fab lines run 24/7 for automotive customers, and her current quality gate catches defects too late -- at final electrical test, after the full processing cost is already in the lot.

The discipline skills: offline/online feature parity as the central engineering challenge of feature architecture; cost-aware feature selection against a sub-second latency budget; schema evolution as an ongoing policy; the training judgment that asks whether ML is even the right approach; shadow deployment as the safe form of A/B in a domain where bad models mean scrapped wafer lots; model versioning and rollback via the registry; blue-green deployment with traffic-shift schedules; graceful degradation to a rule-based fallback; pipeline orchestration with an explicit automation boundary, failure paths, and concurrent-experiment protocol; SLOs and error budgets that govern the lifecycle; an IATF 16949 audit-ready handover pack written for two readers -- the operator and the auditor.

The AI-direction lesson: a production ML system is an integrated architecture, not a deployed model, and AI cannot hold the trade-off interactions across feature parity, scaling, cost, degradation, and deployment simultaneously. You design the architecture and the schemas; AI builds within them. The register is architecture-as-direction at capstone scale, and the infrastructure is now load-bearing -- poor project memory, the wrong hook tier, a fail-open hook all produce bad sessions, and you will experience this directly.

Your Role

Production architect for the defect-prediction system. Sub-second latency, 99.9% availability during fab operating hours, IATF 16949 traceability for every automated decision, product-line-specific quality thresholds, and the augment-not-replace constraint -- operators retain authority over scrap and rework -- shape every decision.

The register shifts again. Last time the architecture spec spanned four pipeline pieces and three custom-agents; this time it spans five production stages -- data, feature, training, serving, monitoring -- bounded by four custom-agents with explicit schemas at every boundary. Marcus Webb is on call again for the serving architecture; you haven't worked with him since the multi-service cloud deployment for Stoffwechsel.

What's New

Last time you ran the four-paradigm comparison for Willem de Vries, with classical, RAG, and LLM generation chosen per pipeline piece. The schema-first decomposition that made composition verifiable there is the working basis here. You ran it across four paradigms there; you'll run it across five production stages here, with monitoring as a distinct stage and the offline/online parity test as the load-bearing composition check.

What's genuinely new: offline/online feature parity as the deepest expression of leakage -- features that look easy in batch aren't computable at prediction time, and the divergence is invisible without an explicit parity test in CI. Shadow deployment as the safe form of A/B. Blue-green traffic-shifts with comparison metrics at every step. Graceful degradation to statistical process control when the ML service is down. SLOs with error budgets that name the exact action when the budget is exhausted. Path-scoped project memory. Hook tiers as a spectrum -- every hook is type: command, and the tier names what the command does: a deterministic script for cheap exact checks, a claude -p call for fast single-prompt judgment, a claude -p --allowedTools agent run for thorough multi-file audit. A portability test in a second AI coding agent. A fourth authored skill, production-architecture-review. An eighteen-ADR series plus a feature catalogue, SLO document, pipeline architecture document, runbooks, and the IATF handover pack.

The hard parts cluster on the new terrain. AI's first feature set will include a batch statistic that isn't computable online. Its first A/B proposal will be random-assignment. Its first deployment plan will stop-then-start. Its first serving design will have no fallback. The discipline rests on catching the default at each juncture and writing the architecture that makes the catch durable.

Tools

• Claude Code -- project memory now path-scoped per pipeline directory; four custom-agents (feature-pipeline-agent, training-pipeline-agent, serving-pipeline-agent, monitoring-pipeline-agent); three hooks across different tiers (all type: command); phase-scoped MCP adapted to four production phases
• scikit-learn + XGBoost / LightGBM + SHAP -- the gradient-boosting model on engineered features (familiar)
• MLflow -- registry stages drive blue-green deployment and rollback (familiar surface, new lifecycle role)
• FastAPI + Docker + uvicorn -- blue-green serving layer with the SPC fallback service alongside (familiar)
• Prometheus + Grafana -- SLO dashboards across quality, drift, cost, safety, latency, with interaction overlays (familiar tools, new architecture)
• Pydantic -- schema validation at every pipeline boundary and at MES ingestion (familiar)
• A second AI coding agent -- for the portability test; AGENTS.md is the bridge artefact (new)
• SKILL.md (agentskills.io) -- the fourth authored skill, production-architecture-review; the three prior skills carry forward
• Python, pytest, Git/GitHub -- pytest extended with the offline/online parity check, schema-conformance at every boundary, and the IATF traceability audit as a pre-promote gate

Materials

You receive:

• first-contact-meeting-transcript.md -- Shu-Fen's structured Zoom briefing
• process-data-sample/ -- about 50,000 wafer records across three fab lines, three months of process parameters with lot IDs, product-line tags, and final-test labels
• aoi-results-sample/ -- per-wafer AOI summary statistics, already in feature-ready form
• electrical-test-sample/ -- the ground-truth labels
• spc-rules-current.yaml -- the current statistical process control rules; both the ML-vs-SPC baseline and the future fallback
• iatf-audit-checklist.md -- what an IATF 16949 auditor expects from automated quality decisions
• schemas/ -- stub directory; five JSON Schema skeleton files you fill during the feature and architecture units
• hook-templates/ -- starting points for the three hook tiers (deterministic / fast-model / deep-agent)
• training-rubric.md, cost-estimator.py -- carried forward, adapted for the ML-vs-SPC comparison and the multi-stage pipeline cost model
• requirements.txt, docker-compose.yml -- the pinned stack and the eight-service skeleton (postgres feature store, MLflow tracking, blue and green prediction services, SPC fallback, routing layer, Prometheus, and Grafana)
• CLAUDE.md -- root project memory with placeholders for the substantial sections you'll add, plus a pointer to path-scoped fragments per pipeline directory
• AGENTS.md -- four custom-agent placeholders
• .mcp-phases.template.md -- phase-scoped MCP template
• skills/_template/ -- the SKILL.md template for the fourth skill