Fine-tuning an Open-Weight Model on a Tight Budget

You're designing a fine-tuned content-generation model for Ama Mensah at Owusu Learning, a social enterprise in Accra building adaptive learning for Ghanaian secondary-school students. Her team has hand-edited about 5,000 curriculum-aligned pieces that off-the-shelf models won't produce on their own.

The discipline skills: parameter-efficient fine-tuning (LoRA and QLoRA) on an open-weight base, base model selection across size and pretraining trade-offs, cloud GPU cost estimation before launch, a dual evaluation that measures the target task and general capability together, serving under a fixed monthly ceiling, and an upfront comparison of prompting, RAG, and fine-tuning so the choice is the one the evidence supports. LoRA and QLoRA are the parameter-efficient methods at the center of this: instead of retraining all of a model's weights, they train small add-on adapter matrices (QLoRA on a 4-bit-quantized base), which is what makes fine-tuning a 9B model fit a tight budget.

The AI-direction lesson: fine-tuning is one approach among three, and the choice is a deliberate professional act. AI will reach for the largest model, the longest training run, the highest adapter rank, and an evaluation that only measures the trained-on task. Each default looks reasonable alone; together they burn the budget and ship a model that has quietly lost capabilities it used to have. The new register is the recipe -- you name the base model, the rank, the data slice, the cost ceiling, and the dual evaluation, and AI executes inside those constraints.

Your Role

Fine-tuning engineer and cost architect on a generation system with a tight inference budget. The $200/month ceiling, the Twi-English code-switching, the pedagogy-quality bar set by Ama's content team, and the one-time-or-rare training cadence shape every decision -- which base model, which adapter rank, which GPU type, which serving option.

The register shifts. You're the architect of the recipe; AI's job is to execute within it. The infrastructure you authored last time -- project memory, the first skill, hooks, phase-scoped MCP -- carries forward as the substrate. The recipe is the direction surface this time; the new terrain is yours. A senior ML engineer is reachable on-demand for one thing only -- whether your catastrophic-forgetting evaluation actually catches forgetting -- and he won't volunteer or hand you answers.

What's New

Last time you built a cloud RAG system for Tariq Al-Rashid at the Jordan Civil Defense Directorate. You ended that project with a diagnostic that said "fix retrieval, not the model." That diagnostic is the bridge into this one.

This time it points the other way. Ama's problem isn't what the model knows; it's how the model writes. RAG can inject Ghanaian examples but the generation register stays American-textbook. Prompting handles surface swaps but not the pedagogy pattern. Fine-tuning is the right answer here -- and proving it with three baselines before any training is the first piece of professional work, not an optional preamble.

The hard parts cluster on the new terrain. AI's first LoRA configuration will use the default rank and trade general capability for target-task improvement without noticing. Its first base-model suggestion will be the biggest open model available, at an inference cost the budget can't carry. Its first training plan will skip the cost estimate. Its first evaluation will only test the fine-tuning data and report a clean win on a model that has narrowed.

Tools

• Claude Code -- project memory extended with a fine-tuning recipe section once it stabilises; the rag-eval skill from last project is the precedent for a new one authored here (familiar, new register)
• Anthropic SDK -- prompting and RAG comparison baselines against Claude Sonnet before any training (familiar)
• Hugging Face transformers + PEFT -- transformers, peft, accelerate, bitsandbytes for QLoRA, trl for the trainer (new)
• A cloud GPU provider -- one of Modal, RunPod, Lambda Labs, or SageMaker, picked on cost; spot pricing, auto-shutdown, and spending alerts (new). The training-phase unit walks through provider setup
• A managed LLM inference API supporting LoRA adapters -- Together AI, Fireworks AI, or similar; chosen on the volume threshold against the $200 ceiling (new). The serving unit walks through setup
• MLflow -- extended to fine-tuning artefacts: base model hash, adapter weights, training data slice, recipe config, eval results, GPU cost (familiar)
• Gemma 2 9B -- the base model, selected in the base-model probe; pinned by hash (new)
• A 100-item MMLU-style sample -- basic reasoning, general science, math, English comprehension; this is what catches catastrophic forgetting
• Python, datasets, pytest, Git/GitHub -- familiar; pytest extended with dataset-structure, tokenisation-preservation, recipe-config, and dual-evaluation checks
• SKILL.md (agentskills.io) -- a second authored skill, eval-fine-tuned-model, bundles target-task eval, general-capability eval, and cost summary into one invocation (familiar pattern, new contents)

Materials

You receive:

• voice-memo-transcript.md -- Ama's voice memo, the first-contact artefact
• curriculum-standards.md -- a one-page summary of the Ghana Education Service Senior High School syllabus
• corpus-sample/ -- about 50 non-confidential hand-edited pieces tagged with subject, topic, difficulty, and language_tag; mixed English-only and Twi-English code-switched
• training-rubric.md -- the target-task rubric: Ghanaian-context register, pedagogy quality, curriculum alignment, code-switching preservation
• general-capability-eval-sample.json -- the 100-item benchmark subset for the catastrophic-forgetting check
• recipe-template.yml -- the structured recipe you fill with base model, rank, target modules, learning rate, epochs, training data slice, eval harness, GPU type, duration estimate, cost estimate
• cost-estimator.py -- a small script you adapt with provider pricing during the training phase
• requirements.txt -- the fine-tuning stack pinned
• CLAUDE.md -- universal rules carried forward, plus an empty "Fine-tuning recipe -- to fill once stabilised" section and a note that the senior colleague on this project is narrow and on-demand only (the catastrophic-forgetting-evaluation seam)
• AGENTS.md -- carried forward with two placeholders (data-prep, training-config)
• skills/_template/ -- the SKILL.md template for the second authored skill