learntodriveai.dev/Data Engineering/Designing a Governance and Quality Architecture Before Any Code
Data Engineering·Project 18·6 units

Designing a Governance and Quality Architecture Before Any Code

**Track:** Data Engineering

§ Brief

Grace Banda at the Malawi National Blood Service emails you with one sentence of scope: privacy must come first, not as an afterthought but as the first design constraint. There is no inherited system, no template, no pipeline code on day one.

The discipline skills: a PII taxonomy written before any model, using the Malawi Data Protection Act's vocabulary alongside the donor-consent axis it does not name; a masking strategy, access model, retention policy, and audit-trail design each derived from the artefact above it; a quality architecture written for the Data Protection Authority with the engineering detail in an appendix; transformation, ingestion, and observability shaped to honour the design package already on paper; an implementation the design has already constrained; a coverage map at the close.

The AI-direction lesson: agents reorganise again — by design surface this time, one agent per artefact, each reading only the upstream artefacts it depends on and writing only on the artefact it authors. The decomposition is the architecture's coherence guarantee. Without it, AI's per-component coherence drifts into per-component contradiction — a masking strategy that refers to categories the taxonomy dropped, an access model that ignores the retention rule. The regulatory-interpretation calls — which Data Protection Act category applies where, whether donor consent for transfusion extends to anonymised statistics, why cold-chain "real-time" is really a 5-minute batch — stay with you.

Your Role

You are the governance architect for the blood service's first design pass. You reply to Grace in her register, open a discovery thread, decompose the design-first frame into a sequence of artefacts, write each in the order the constraints demand, bring the taxonomy and the masking strategy to Priya Venkatesh for review, implement against contracts you have already signed off, and deliver Grace a working system plus a regulator-facing compliance package the Data Protection Authority can read.

The scaffolding is thin. No taxonomy template, no masking template, no access-model template, no quality-architecture template. Grace names a principle; you decompose it.

The AI relationship sits at autonomous. You design the decomposition before any agent is asked to draft anything, scope each agent tightly to what it needs to read and what it is allowed to write, and hold the constraint flow yourself — the upstream artefact is signed off before the downstream agent is even configured.

What's New

Last project you audited SurBauxiet's inherited warehouse against a ministry memorandum — forensic work on a system that already ran, agents organised by work role, no senior colleague.

This project nothing runs on day one. Genuinely new: the PII taxonomy as the first artefact written; a constraint-flow sequence where each artefact carries forward as named constraints into the next; per-design-surface agents in place of per-work-role; Priya Venkatesh back to review the taxonomy and the masking strategy where her calibration shapes everything downstream; a quality architecture written in regulator vocabulary before any dbt model exists; a coverage map tying every taxonomy category, access role, retention rule, and audit invariant to a verification surface.

The hard part: holding the design-first discipline when AI is fluent and the implementation tools are familiar. Asked to "set up the dbt project," AI will scaffold one, write a staging model, and apply a generic hash to anything that looks like PII. You refuse until the taxonomy is reviewed and the masking strategy is named.

Tools

  • dbt, Soda Core, BigQuery, Dagster, GitHub Actions, Claude Code, Codex CLI, MCP, path-scoped CLAUDE.md — all carry-forward. No new tools.
  • The distinctive practice is the per-design-surface agent organisation: one agent per artefact — taxonomy, masking, access, retention, audit, quality, observability — each with read scope only on its upstream dependencies and write scope only on its own artefact. The third agent-organisation axis after P16 (per-consumer-domain) and P17 (per-work-role).
  • Codex CLI runs the cross-check pass on the taxonomy and the masking strategy — a second-tool reading to catch the cross-document coherence AI is not reliably good at.

Materials

You'll receive:

  • A project governance file (CLAUDE.md) — design-first context: client, the design-first principle, the constraint-flow sequence, the per-design-surface agent organisation, verification targets, commit convention.
  • Blood-service operational picture (blood-service-context.md) — what the service does layer by layer: collection at centres and mobile teams, infectious-disease testing, processing into components, cold-chain storage, distribution to 50+ hospitals, donor-to-unit-to-patient traceability.
  • Malawi Data Protection Act extract (malawi-dpa-extract.md) — the Act's relevant sections in its own language. The taxonomy designer reads this directly.
  • Discovery prompt sheet (discovery-prompts.md) — starter targeted questions for the thread with Grace. The hidden constraints surface only on questions that ask after them.
  • A source seed script (scripts/seed-mbts-sources.py) — representative source data across donors, donations, test results, processing, cold-chain readings, distribution, and transfusions. Held in reserve through Units 1-4; not invoked until the design package is signed off.

Deliberately not provided: any template for the taxonomy, masking strategy, access model, retention policy, audit-trail design, quality architecture, or compliance package. The SurBauxiet workspace does not carry forward — fresh project root.