You're designing a hybrid production system for Ana Beatriz Costa at Banco Horizonte, a São Paulo retail bank. One stack, two paradigms: a classical XGBoost fraud model on a 50-millisecond synchronous path and an LLM customer-service component on the WhatsApp message path, with a routing layer in front and a shared fraud-flag store between them.
The discipline skills: real-time feature serving with a cache, training-serving parity, intent-aware retrieval into the LLM context, LLM serving guardrails (confidence-threshold escalation, structured output schema), a routing layer with cost-aware cascading, multi-service orchestration, per-segment evaluation across customer registers.
The AI-direction lesson: verification across paradigms. Catching AI is three different acts here. Inside the fraud component (recompute aggregates per request, training-serving drift) it is classical-ML verification. Inside the LLM component (no escalation threshold, fraud-flag buried in the prompt) it is LLM verification. At the integration boundary (synchronous chain that breaks the latency ceiling, single service that starves the fast path) it is system verification. Same stance, three techniques, all designed before any code runs.
Your Role
Hybrid-system architect on a regulatory-grade production stack. The 50ms fraud ceiling, escalation threshold, and audit log are Banco Central regulations; the R$50,000/month budget is a hard CFO commitment. You handle the architecture call, the routing rules, the per-component latency budgets, the LLM guardrails, the flag-store data flow, and the system-level evaluation that catches what component metrics hide.
The scaffolding holds at P16's shape: no primer files, and the senior colleague narrowed to a single seam. Independent across the established directing surface and each discipline-column pattern carried in. The new terrain is the system architecture itself. A senior ML-systems-and-compliance consultant is reachable on-demand for one thing only — whether the routed system holds at its seams and survives Banco Central scrutiny — and he won't volunteer or hand you answers.
What's New
Last time you built a property-valuation bridge for Kerem Yilmaz at Emlak Vizyon -- embeddings as features for a classical regressor, an honest ablation, a version-pinned feature store, and your first delegated agents with scope boundaries. Delegation by design and verification at composition boundaries are floor.
This time the components don't compose into one model -- they live in one system with a routing layer between them. The fraud model writes structured flag records; the LLM reads them when routing tags a query as fraud-related. The routing decision is the architecture, and the architecture is what makes agent scopes, verification design, and context boundaries tractable. Three custom agents, one per component. One MLflow MCP server covers both paradigms.
The hard parts are scoped quietly. AI's first system architecture will be a synchronous chain end-to-end, breaking the latency ceiling on every fraud query. Its first LLM code will skip the escalation threshold -- a regulatory failure, not a quality regression. Its first cost analysis will route every query through the LLM and exceed the budget. Its first monitoring will split fraud and LLM panels with no system view. Each catch is a different verification technique.
Tools
- Claude Code -- primary AI agent; project memory extended with hybrid-system rules; three custom agents in
.claude/agents/, each scoped to a component (familiar, deeper) - Anthropic SDK -- the LLM customer-service component runs against Claude Haiku (familiar)
- Python / pandas -- transactions, queries, monitoring (familiar)
- scikit-learn -- classical intent classifier in the cascade (familiar)
- XGBoost -- the existing fraud model; integrate, not retrain (familiar)
- FastAPI -- two services on separate processes (familiar)
- Pydantic -- structured output schema for fraud explanations (familiar)
- Redis -- feature-store cache for precomputed fraud features (new)
- MLflow -- classical and LLM evaluation via MCP server (familiar)
- Docker Compose -- four services with separate health checks (familiar, deeper)
- pytest -- extended with system-integration, parity, routing, and regulatory-logging tests (familiar)
- Prometheus + Streamlit -- monitoring layer with a multi-dimensional dashboard (new at this register)
- Git/GitHub -- familiar
The unit that introduces each new tool walks through setup.
Materials
You receive:
- An
email-chain.md-- Ana's forwarded threads from her fraud and customer-service leads, with annotations naming the budget, latency ceiling, and regulatory constraint - A
regulatory-requirements.md-- the Banco Central regulations in plain language - A
transactions/directory -- ~250,000 synthetic transactions with fraud labels, the XGBoost artefact, and the feature spec - A
customer-queries/directory -- ~5,000 synthetic Brazilian-Portuguese WhatsApp queries with intent, register, and segment tags, plus a held-out subset - A
fraud-flag-records/directory -- JSON Schema, sample records, and an evolution rule - A
routing_rules_template.yml-- empty structured artefact with a schema header CLAUDE.mdandAGENTS.mdcarried forward -- bridge rules generalised, a labelled empty "Hybrid-system-specific rules" section, three agent placeholders- A
docker-compose.ymlskeleton with four services (feature-store,fraud-scoring,llm-customer-service,monitoring) and TODO integration points - Three agent templates in
.claude/agents/, plus a.env.exampleforANTHROPIC_API_KEY
The senior-colleague handle is narrow: an on-demand systems-and-compliance consultant for the integration-boundary seam only. For everything else, when you're stuck, you ask AI directly.