RAG Pipelines: Hybrid Retrieval, Generation, and Groundedness Verification.

You're building a retrieval-augmented generation system over a clinical-trial document archive for Yuki Nakamura, Director of Regulatory Documentation at a Tokyo pharmaceutical company. The system has to retrieve from a versioned, bilingual corpus and generate answers her compliance team can cite in PMDA submissions.

The discipline skills: hybrid retrieval combining vector similarity with keyword search (with proper score normalisation), embedding-model selection driven by held-out evaluation, context-window assembly with explicit ordering and redundancy controls, generation with a grounding system prompt, three-failure-mode evaluation (retrieval, context assembly, generation), groundedness verification, and per-query cost tracking on a paid hosted API.

The AI-direction lesson: the directing relationship now spans two AI layers -- the AI that writes your code, and the LLM whose output that code produces. Both have defaults to catch. AI's first RAG pipeline averages BM25 and cosine scores without normalising (so "hybrid" is keyword-only), evaluates end-to-end (which hides which component failed), writes a grounding prompt the LLM ignores under partial context, hardcodes API keys, and ships paid calls with no cost tracking. The verification habit carries; what changes is that you verify both layers, and the test plan has to distinguish three different failure modes.

Your Role

ML engineer building your first RAG pipeline on regulatory-grade terrain. Yuki's compliance team has zero tolerance for unsourced claims -- a hallucinated citation in a PMDA submission is a compliance violation, not a bug. You handle the retrieval upgrade, context assembly, generation, groundedness verification, and cost monitoring on the first paid API call.

The directing infrastructure carries forward unchanged from P14: project memory, meta-prompting, decomposition discipline, full-discovery pacing. What's independent is everything you brought from P14 -- chunking, embeddings, vector retrieval, retrieval evaluation, behavioural tests. What's guided is the terrain stacked on top: RAG architecture, three failure modes, groundedness verification, paid API integration. The chunker, embedder, and index are the floor, not the project.

What's New

Last time you built a semantic search system for Diego Fuentes -- chunking, embeddings, ChromaDB, precision@k and recall@k against a held-out query set, and a local LLM through an OpenAI-compatible API for chunk summarisation. Retrieval is now floor.

This time you stack generation on top of retrieval and discover the system has three distinct failure modes, not one. New work: hybrid search (vector plus BM25, with the normalisation AI keeps skipping), context-window assembly as a data-quality decision, a grounding prompt that has to hold under partial context, and groundedness verification as the central new act. Cost becomes real too -- this is the first paid API call in the track.

The hard part is decomposed evaluation. A response that reads "According to the Q3 protocol, the safety threshold is 5mg" when the protocol says 50mg is worse than no answer -- Yuki's team will trust the citation. End-to-end evaluation cannot tell you whether retrieval, context assembly, or generation failed. You design three separate test strategies and run them against a held-out set Yuki authors with you. The test plan is the deliverable that makes failures visible.

Tools

• Claude Code -- primary AI agent, project memory carried forward (familiar)
• Python / pandas -- corpus inspection, eval-set handling, cost reporting (familiar)
• LangChain or LlamaIndex -- document loaders and text splitters (familiar)
• ChromaDB -- local-disk vector database, extended here with metadata filtering for version-aware retrieval (familiar)
• sentence-transformers -- embedding model library, with MTEB-driven selection and a multilingual variant for Japanese/English (familiar, deeper)
• rank_bm25 -- keyword retrieval, paired with vector search for hybrid retrieval (new)
• Anthropic Python SDK -- the first paid hosted API in the track; Claude Haiku for cost-conscious generation (new)
• Ollama -- local LLM serving, used for prototype generation and the local-vs-hosted cost baseline (familiar)
• promptfoo -- LLM evaluation framework for behavioural and groundedness test cases (new)
• pytest -- extended with groundedness and cost-tracking tests (familiar)
• Git/GitHub -- familiar

The unit that introduces each new tool walks through setup.

Materials

You receive:

• A clinical-trials-corpus/ of mock pharmaceutical documents -- versioned protocols, case report forms, investigator brochures, and regulatory correspondence in mixed Japanese and English, with a version-history sub-structure
• Single-page primers for the new vocabulary: hybrid-search-primer.md, rag-primer.md, groundedness-primer.md, context-window-primer.md, cost-monitoring-primer.md
• An eval-seed/ with three seed queries from Yuki and a JSON template for the held-out test set you build with her (Yuki delivers the populated groundedness-test-cases.yaml later, during Unit 4)
• Your CLAUDE.md and AGENTS.md carried forward, with an empty section "RAG-specific rules -- to extend during this project"
• A docker-compose stack for Ollama, used as the local cost baseline
• A .env.example showing how to wire the Anthropic API key without committing it