Approach Selection Across Modelling Paradigms

You're designing a four-piece claims-processing pipeline for Willem de Vries, VP of Claims Operations at Verzekeraar Digitaal, a Dutch insurance company in Amsterdam. His team currently does everything manually -- categorise the claim, score it for fraud, match it against policy clauses, draft the decision letter. He has been told AI can help and is wary of "one hammer for every nail" projects.

The discipline skills: approach selection across classical ML, RAG, and LLM generation -- chosen per pipeline piece against accuracy, cost, latency, and explainability; hierarchical retrieval over a versioned policy corpus with citation tracking built into the architecture; safety guardrails and contradiction handling tuned to the domain; a fairness audit that names which definition was chosen and why; multi-agent decomposition with explicit schemas at every boundary; model cards written for two readers -- Willem and the Dutch DPA.

The AI-direction lesson: approach is not one paradigm for the whole project. It's one paradigm per problem piece, chosen against multi-objective constraints that conflict. AI's default is one large model for everything, optimised for accuracy alone -- flat retrieval over a structured corpus, confidence-based routing where confidence isn't calibrated, citations without verification, a single fairness metric without naming the trade. The new register is architecture-as-direction: you name the four-paradigm decomposition, the multi-objective scoring surface, the fairness definition, the agent schemas, the guardrail thresholds. AI executes inside that spec.

Your Role

Approach-selection architect for a regulated insurance pipeline. GDPR Article 22, the Dutch DPA's audit expectations, the 14-days-to-5-days processing goal, the 80,000-claims-per-year volume, the ~500ms latency budget per fraud decision, and the augment-not-replace constraint shape every decision.

The register shifts again. Last time the recipe was the direction surface; this time the architecture spec is -- four pipeline pieces, three custom-agents bounded by JSON schemas, and a fairness decision that travels to the regulator. AI's job is execution within the architecture. Dr. Sarah Chen is on call again for the fairness-architecture work; you haven't worked with her since Hayashi Pharmaceutical.

What's New

Last time you fine-tuned an open-weight model for Ama Mensah on a tight budget, with the prompting/RAG/fine-tuning comparison run before any training. That comparison vocabulary -- "prompting solves behaviour, RAG solves knowledge, fine-tuning solves consistency" -- is the bridge into this one. You ran it on one problem there. You'll run it on four here.

What's genuinely new: multi-objective optimisation with the Pareto frontier as the decision surface, not a single ranking. Hierarchical retrieval that mirrors how a claims handler actually searches, with citation tracking built in so every policy reference resolves back to a real clause. Fairness definitions that are mathematically incompatible -- demographic parity, equalised odds, predictive parity can't all hold at once; choosing among them is a values decision you communicate to two audiences. Multi-agent decomposition where three agents produce locally correct output that doesn't compose globally, until the schema at every boundary becomes the contract. Async delegation to the GitHub Copilot coding agent for bounded sub-tasks. Model cards as the artefact that accompanies each piece into production.

The hard parts cluster on the new terrain. AI's first proposal will be "fine-tune one LLM for all of it." Its first fraud-scoring comparison will report F1 only. Its first cascading router will be confidence-based. Its first retrieval will be flat. Its first multi-agent decomposition will leave the boundaries implicit. The discipline rests on catching the default at each juncture and writing the architecture that makes the catch durable.

Tools

• Claude Code -- project memory grows substantially; three custom-agents authored (classification-config, retrieval-config, generation-guardrails); phase-scoped MCP adapted to four phases
• Anthropic SDK -- the prompting baseline and the LLM decision drafter in production (familiar)
• scikit-learn + XGBoost / LightGBM + SHAP -- the classical fraud scorer with feature-level explainability for the GDPR audit trail (familiar)
• Hugging Face transformers + sentence-transformers + pgvector -- the hierarchical RAG over the policy corpus (familiar surface, new architecture)
• FastAPI + Docker + MLflow -- two production endpoints and a tracking server (familiar)
• Fairlearn (or equivalent) -- demographic parity, equalised odds, predictive parity computed across protected attributes (new). The fairness-audit unit walks through setup
• GitHub Copilot coding agent -- async delegation for two bounded sub-tasks specified as GitHub issues with acceptance criteria (new)
• SKILL.md (agentskills.io) -- a third authored skill, approach-decision, encoding the multi-objective comparison workflow; rag-eval and eval-fine-tuned-model are the precedents (familiar pattern, new contents)
• Python, pytest, Git/GitHub -- pytest extended with schema-conformance checks at agent boundaries, citation-accuracy checks, and disaggregated retrieval evaluation

Materials

You receive:

• first-contact-email.md -- Willem's terse direct email
• claims-data-sample/ -- about 2,000 historical claims with categorisation labels, fraud flags, amounts, policy IDs, and simulated protected attributes
• policy-corpus-sample/ -- about ten policy documents across property, auto, and liability, in both 2024 and 2025 versions
• schemas/ -- empty; you fill it during the architecture unit
• training-rubric.md, general-capability-eval-template.json -- rubric and table structure for the per-piece approach comparison
• fairness-toolkit-example.py -- a minimal script computing the three fairness definitions, adapted during the audit
• citation-verification-template.py -- a template you either fill yourself or assign to the Copilot coding agent
• model-card-template.md -- the template for the four model cards you write at handover
• cost-estimator.py -- extended for cost-per-claim averaging across the cascading architecture
• operations-runbook-template.md -- the monitoring-to-action mapping the runbook will hold
• docker-compose.yml, requirements.txt -- the four-service skeleton and the deliberately compatibility-coordinated multi-paradigm stack (the requirements header explains why the pinned set is not bumped piecemeal even when newer releases exist)
• CLAUDE.md -- universal rules carried forward, with placeholder sections labelled for the substantial content you'll add
• AGENTS.md -- three custom-agent placeholders
• .mcp-phases.template.md -- phase-scoped MCP template for the four P20 phases
• skills/_template/, tests/coherence/ -- the SKILL.md template and the coherence-test directory, which ships with designed-in failure fixtures you encounter during multi-agent integration