learntodriveai.dev/Data Engineering/Operating an Inherited Warehouse Under a Live Regulator Request
Data Engineering·Project 17·5 units

Operating an Inherited Warehouse Under a Live Regulator Request

**Track:** Data Engineering

§ Brief

Henk Pengel at SurBauxiet N.V. inherits you onto a running warehouse and a Ministry of Natural Resources compliance memorandum with a 48-hour quarterly reporting clock.

The discipline skills: governance audited against an external regulator request rather than designed from scratch; SLOs renegotiated against weeks of burn-rate evidence; observability debt identified and retired as a deliverable; retention enforcement verified by querying the live retention-bound tables, not the policy document; a ministry-facing compliance report and an engineer-facing operations-debt register written from the same audit.

The AI-direction lesson: agents reorganise by work role — an audit-only agent that reads every data product but writes nowhere, a remediation agent scoped to the specific files it touches, a ministry-report agent that reads findings and writes only the report. AI handles the mechanical work. The regulatory-interpretation calls (is calibration staleness a governance finding or a quality finding under ministry rules? is the concession-area code overlap a schema issue or an operational one?) stay with you. "When NOT to delegate" is the project's defining performance.

Your Role

You are the operations data engineer for SurBauxiet's inherited warehouse. You read the memorandum, open a discovery thread with Henk, audit the governance and observability state against the ministry's request, plan and execute a remediation that respects the live system, and deliver two artefacts — the ministry-facing compliance report and the engineer-facing operations-debt register — alongside maintained infrastructure the next operator can pick up.

The scaffolding is thin. No audit template, no remediation template, no compliance-report template, and no colleague who reviews the work. Henk hands over a one-page memorandum and a deadline; everything else is yours. (Roderick Amattaram, a regulatory-compliance advisor, is reachable on demand for a narrow sanity check on whether a finding is framed the way the ministry's own control framework would frame it — he volunteers nothing and runs no part of the audit.)

The AI relationship sits at autonomous. You decompose audit-and-remediation work for a system that already exists, supply each work-role agent with the constraints its scope demands, and verify that findings, remediation changes, and report claims compose into one coherent operational story.

What's New

Last project you designed the warehouse architecture for Nordfjord — four consumer groups, integrated quality and governance and observability as one system, per-consumer-domain agents, a board-facing cost forecast.

This project the architect-designing-from-scratch role disappears. The warehouse is described to you as it operationally is. Genuinely new: a formal memorandum from a regulator-facing client; forensic-then-corrective work rather than architectural-then-implemented; agents organised by work role (audit / remediate / report) with the audit agent's read-only stance a tighter connectivity discipline than anything you have configured; two reports from one audit because the audiences — ministry and next operator — speak different languages; infrastructure maintenance as part of the deliverable.

The hard part: operational drift hides in the gap between what the architecture documents say and what the live system actually does. A retention rule passes its CI gate while a row that should have been deleted six weeks ago is still there. An SLO reports green for weeks while the consumer complains. A flaky test gets muted. A masking macro is correct everywhere except the new column an upstream team added last month. Your job is to find what the warehouse has not announced.

Tools

  • dbt, Soda Core, BigQuery, Dagster, GitHub Actions, Claude Code, Codex CLI, MCP, path-scoped CLAUDE.md — all carry-forward. You choose from the full toolkit; no new tools.
  • The distinctive practice is a deliberate split between an audit-only agent (read everywhere, write nowhere) and a remediation agent (write only on the specific retention macros, SLO definitions, and CI files the remediation touches). Selective MCP connectivity is familiar; the scoping discipline is new.
  • BigQuery INFORMATION_SCHEMA.JOBS and the existing observability dashboards are read forensically — you are reading what weeks of operation have produced, not designing what to monitor.

Materials

You'll receive:

  • A project governance file (CLAUDE.md) — SurBauxiet operational-audit context: client, inherited warehouse as Henk's office has it on record, ministry compliance request, audit-and-remediation scope, verification targets, commit convention.
  • Inherited warehouse state (inherited-warehouse-state.md) — data products that exist, SLOs running, retention rules in force, masking and audit-trail conventions. Operational record, not architecture documentation.
  • Ministry request pack (ministry-request-pack.md) — the regulator's specific request in the regulatory vocabulary you must respond to: freshness commitments for environmental monitoring, access-audit trails, retention windows for worker medical records, production by concession area.
  • Audit prompt sheet (audit-prompts.md) — starter targeted questions for the discovery thread. Hidden constraints surface only on questions that ask after them.
  • Warehouse seed script (scripts/seed-surbauxiet-warehouse.py) — generates a representative seed as local CSV/JSON you load into BigQuery if you want a runnable system to audit against.
  • Henk's formal memorandumSB-2026-047, in the chat at project start.

Deliberately not provided: any template for the audit report, remediation plan, compliance report, or operations-debt register. The Nordfjord workspace does not carry forward — different client, different domain, fresh project root.