You're building a semantic search system over a corpus of legal documents for Diego Fuentes, the managing partner of a small commercial law firm in Santiago. Lawyers will type plain-language questions and the system has to surface the right precedents -- not keyword matches.
The discipline skills: chunking documents into retrieval-sized pieces with metadata that makes citation possible, embedding text into vectors and indexing them in a local vector database, designing a small ground-truth eval set with the client and computing precision@k and recall@k against it, serving a local LLM through an OpenAI-compatible API, and writing behavioural tests for the LLM layer (refusal, citation, register).
The AI-direction lesson: you've spent thirteen projects on classical ML -- features, training, metrics, deployment. This is a different shape of problem. There's no training, no train/test split, no hyperparameters. AI knows that, but its defaults on a fresh retrieval pipeline are still wrong in specific ways. It chunks documents at fixed character counts and splits sentences mid-clause. It strips chunk metadata, so the system can't tell you which document an answer came from. It ships pipelines with no retrieval evaluation -- just retriever.query(...) and a list of plausible-looking results. It defaults to OpenAI's hosted endpoint with a hardcoded API key, which is the wrong answer for documents under client privilege. The directing skills you've built carry; the surface they're applied to is new.
Your Role
NLP engineer entering retrieval terrain for the first time. Diego is skeptical -- three previous document systems have failed him -- and he chose a small engagement deliberately. The system runs on the firm's office hardware, not in the cloud, because privileged documents do not leave the building. You handle corpus prep, chunking, embeddings, indexing, eval, and the local LLM layer.
The scaffolding stays at full discovery: a single email from Diego, no forwarded chain, no senior colleague brief setting up the work. The directing infrastructure you've built across the track -- project memory, meta-prompting, decomposition discipline -- carries forward unchanged. What's introductory is the LLM-and-retrieval terrain itself.
What's New
Last time you carried BridgeCredit's classical ML stack into operations: model registry, automated retraining, A/B testing with significance, and multi-MCP permission design.
This time the entire toolchain is different. No MLflow, no scikit-learn model, no FastAPI scoring endpoint to extend. You're working on documents instead of tabular features, embeddings instead of trained models, retrieval metrics instead of classification metrics, and a local LLM instead of a deployed classifier. The discipline is the same -- data preparation determines outcomes, evaluation must be honest, AI's defaults need catching -- but the terrain looks nothing like what came before.
The hard part is judgment in a new vocabulary. You will see embeddings, vector similarity, semantic versus fixed chunking, top-k retrieval, precision@k and recall@k, OpenAI-compatible APIs, and Ollama for the first time. AI will produce code that looks fine and fails quietly: chunks that split a paragraph mid-sentence, retrievals that look plausible but are wrong, an LLM that answers confidently from the wrong source. The eval set you build with Diego is what makes those failures visible.
Tools
- Claude Code -- primary AI agent, project memory carried forward (familiar)
- Python / pandas -- corpus inspection and eval-set CSV handling (familiar)
- LangChain or LlamaIndex -- document loaders and text splitters (new use, light touch -- not "the framework," just the primitives)
- ChromaDB -- local-disk vector database, embedded as a Python library (new)
- sentence-transformers -- embedding model library;
all-MiniLM-L6-v2is the default (new) - Ollama -- local LLM serving so privileged documents stay on the firm's hardware (new)
- OpenAI-compatible Python client -- pointed at the local Ollama endpoint, then briefly at OpenAI's hosted endpoint to feel the standard (new use)
- FastAPI -- a thin retrieval API endpoint (familiar)
- Docker / docker-compose -- Ollama ships as a container in the materials (familiar)
- pytest -- extended with retrieval evaluation tests and behavioural LLM tests (familiar)
- Git/GitHub -- familiar
The unit that introduces each new tool walks through setup.
Materials
You receive:
- A sample corpus of about 150 mock legal documents under
corpus/-- contracts, briefs, court filings, and regulatory opinions in Spanish and English, with realistic OCR artefacts on a portion of the scanned files - Short single-page primers for the new vocabulary:
embeddings-primer.md,chunking-primer.md,retrieval-eval-primer.md,ollama-primer.md - A docker-compose stack for running Ollama locally
- An
eval-seed/directory with three legally-grounded test queries Diego has signed off on, which you extend into a fuller ground-truth set during the project - Your
CLAUDE.mdandAGENTS.mdcarried forward, ready to extend with retrieval-specific rules - The
mcp-trust.mdfrom last project -- no new MCP servers this time, but the file stays in the project memory