learntodriveai.dev/Data Engineering/Designing a Governance and Quality Architecture Before Any Code
Data Engineering·Project 18·6 units

Designing a Governance and Quality Architecture Before Any Code

**Track:** Data Engineering

§ Brief

Grace Banda at the Malawi National Blood Service emails you with one sentence of scope: privacy must come first, not as an afterthought but as the first design constraint. There is no inherited system, no template, no pipeline code on day one.

The discipline skills: a PII taxonomy written before any model, using the Malawi Data Protection Act's vocabulary alongside the donor-consent axis it does not name; a masking strategy, access model, retention policy, and audit-trail design each derived from the artifact above it; a quality architecture written for the Data Protection Authority with the engineering detail in an appendix; transformation, ingestion, and observability shaped to honor the design package already on paper; an implementation the design has already constrained; a coverage map at the close.

The AI-direction lesson: agents reorganise again — by design surface this time, one agent per artifact, each reading only the upstream artefacts it depends on and writing only on the artifact it authors. The decomposition is the architecture's coherence guarantee. Without it, AI's per-component coherence drifts into per-component contradiction — a masking strategy that refers to categories the taxonomy dropped, an access model that ignores the retention rule. The regulatory-interpretation calls — which Data Protection Act category applies where, what donor consent actually covers, why cold-chain "real-time" is really a 5-minute batch — stay with you.

Your Role

You are the governance architect for the blood service's first design pass. You reply to Grace in her register, open a discovery thread, decompose the design-first frame into a sequence of artefacts, write each in the order the constraints demand, bring the taxonomy and the masking strategy to Priya Venkatesh for review, implement against contracts you have already signed off, and deliver Grace a working system plus a regulator-facing compliance package the Data Protection Authority can read.

The scaffolding is thin. No taxonomy template, no masking template, no access-model template, no quality-architecture template. Grace names a principle; you decompose it.

The AI relationship sits at autonomous. You design the decomposition before any agent is asked to draft anything, scope each agent tightly to what it needs to read and what it is allowed to write, and hold the constraint flow yourself — the upstream artifact is signed off before the downstream agent is even configured.

What's New

Last project you audited SurBauxiet's inherited warehouse against a ministry memorandum — forensic work on a system that already ran, agents organized by work role, no senior colleague.

This project nothing runs on day one. Genuinely new: the PII taxonomy as the first artifact written; a constraint-flow sequence where each artifact carries forward as named constraints into the next; per-design-surface agents in place of per-work-role; Priya Venkatesh back to review the taxonomy and the masking strategy where her calibration shapes everything downstream; a quality architecture written in regulator vocabulary before any dbt model exists; a coverage map tying every taxonomy category, access role, retention rule, and audit invariant to a verification surface.

The hard part: holding the design-first discipline when AI is fluent and the implementation tools are familiar. Asked to "set up the dbt project," AI will scaffold one, write a staging model, and apply a generic hash to anything that looks like PII. You refuse until the taxonomy is reviewed and the masking strategy is named.

Tools

  • dbt, Soda Core, BigQuery, Dagster, GitHub Actions, Claude Code, Codex CLI, MCP, path-scoped CLAUDE.md — all carry-forward. No new tools.
  • The distinctive practice is the per-design-surface agent organization: one agent per artifact — taxonomy, masking, access, retention, audit, quality, observability — each with read scope only on its upstream dependencies and write scope only on its own artifact. The third agent-organization axis after P16 (per-consumer-domain) and P17 (per-work-role).
  • Codex CLI runs the cross-check pass on the taxonomy and the masking strategy — a second-tool reading to catch the cross-document coherence AI is not reliably good at.

Materials

You'll receive:

  • A project governance file (CLAUDE.md) — design-first context: client, the design-first principle, the constraint-flow sequence, the per-design-surface agent organization, verification targets, commit convention.
  • Blood-service operational picture (blood-service-context.md) — what the service does layer by layer: collection at centres and mobile teams, infectious-disease testing, processing into components, cold-chain storage, distribution to 50+ hospitals, donor-to-unit-to-patient traceability.
  • Malawi Data Protection Act extract (malawi-dpa-extract.md) — the Act's relevant sections in its own language. The taxonomy designer reads this directly.
  • Discovery prompt sheet (discovery-prompts.md) — starter targeted questions for the thread with Grace. The hidden constraints surface only on questions that ask after them.
  • Source seed data (data/mbts_raw/) — seven CSVs of representative source data across donors, donations, test results, processing, cold-chain readings, distribution, and transfusions. Held in reserve through Units 1-4; not loaded until the design package is signed off.

Deliberately not provided: any template for the taxonomy, masking strategy, access model, retention policy, audit-trail design, quality architecture, or compliance package. The SurBauxiet workspace does not carry forward — fresh project root.