learntodriveai.dev/Data Engineering/Capstone: One Platform, Four Masters
Data Engineering·Project 24·9 units

Capstone: One Platform, Four Masters.

**Track:** Data Engineering

§ Brief

Claire Donovan is Chief Digital Officer of Yandari Resources Limited, an ASX-listed mid-tier gold and copper miner in Perth with three remote mine sites. She answers to four masters whose needs do not agree, and "one platform, four views" is the entire spec. You're designing and building that platform.

The discipline skills: turning a 14-minute meeting transcript into a defensible architecture; profiling raw exports from three sites whose systems do not match; designing data product boundaries and a coordinated schema-evolution path for four audiences whose contracts move on different cadences; deciding the batch-versus-streaming split per source upfront; building ingestion, a layered transformation, and a right-sized quality architecture; designing an integrated dual-framework governance system; building observability and forecasting cost growth; and standing up the portable AI infrastructure the build runs on. Every skill from twenty-three projects, together, with the added load that the architecture must reconcile a conflict, not just serve a need.

The AI-direction lesson: AI builds each component competently in isolation but cannot hold four-audience coherence. Given "design the data platform," it produces a uniform architecture that treats every source and audience the same, reaches for streaming whenever "freshness" appears, changes a shared dimension without checking which audience contract depends on it, and reports the work done when it runs rather than when the numbers reconcile. The capstone judgment is knowing when not to delegate, and designing a decomposition that still works when an agent gets something wrong — because no one will catch it if you do not.

Your Role

You are the platform architect. By the end you will have a conflict-aware problem statement, a defended four-view architecture, a built and deployed multi-source platform with quality and observability designed in, an integrated governance system, a cost forecast, and a handover another engineer can operate with a different AI tool.

This is the scaffolding floor — lower than anything before it. There is no brief, no spec, no source inventory, no data dictionary, and no colleague who reviews the work. The transcript is the input. (One narrow exception, and it is not a reviewer: Bill Nguyen is reachable on demand for a sanity check on whether two audiences' numbers reconcile — he volunteers nothing, reviews nothing, and decides nothing for you.) Discovering what the three sites hold, what the four audiences actually need, and what is regulated or confidential is the architectural work, not a preamble to it. The judgment about how the pieces fit, whether the platform satisfies four masters at once, and whether it is right rather than merely running, stays with you — and you are the only check.

What's New

Last project you took Faisal Al-Harbi's one-paragraph mandate at Al-Jubail Chemical Industries and architected a full platform from scratch — but for one effective audience, with a senior colleague to sanity-check the work, and a single streaming path retrofitted late.

This removes the colleague and replaces the single audience with four whose needs genuinely conflict: regulators want auditable quarterly truth, investors want a different reporting framework, state environmental agencies want sensor data within a freshness floor, operations wants real-time safety alerts. All served from one platform with four access-controlled views. Batch and streaming are designed together from the first decision, not bolted on. The governance constraint is a dual regulatory framework plus worker visa and medical data.

The hard part is the conflict itself. Four audiences mean four ways for numbers to be silently wrong, and the architecture has to absorb a requirement change at a seam you designed rather than one you discover. Holding the coherence of a four-master platform, from a transcript, with no one to check the work, is the whole test.

Tools

No new tools. Every tool here is one you already know, now composed into a single multi-audience system you architect end to end.

  • dbt — the transformation architecture for four audience-facing data products, with maintainability and backfill designed in.
  • DuckDB or BigQuery — the warehouse. You choose the engine and defend it; also the basis for cross-site reconciliation and the cost forecast.
  • Dagster — orchestration and observability: asset lineage, schedules, sensors, freshness policies, the per-source batch/streaming split.
  • A streaming/CDC path — designed in from the start for the sources whose freshness genuinely demands it, not retrofitted.
  • Soda Core or dbt tests — the right-sized quality architecture, built to survive a backfill.
  • Git and GitHub — the project repository; commit history is the decision record. Final push and operator README.
  • Claude Code — directing the whole build. You design the context lifecycle, the handoff-ready and security-evaluated connectivity, the failure-robust decomposition, and the portable AI infrastructure.

Materials

This is the scaffolding floor. You receive almost nothing authored:

  • The raw multi-site source data (source-data/) — what the three mines actually hold: production, environmental-sensor, financial, and HR/safety extracts. One site's export is from a different vendor system with field names in another language. There is no data dictionary, no README, no manifest. You discover the shape by profiling it, the way a real platform architect receives data.
  • The engagement context file (CLAUDE.md) — a near-empty stub: the client, the tech stack, the commit convention, and an explicit note that the data dictionary, the data product boundaries, the governance rules, and the verification targets are yours to build into this file across the project. Its emptiness is deliberate.

The transcript arrives in chat as Claire's first message. Everything else — the problem statement, the architecture, the dbt and Dagster projects, the quality config, the governance system, the cost forecast, and the handover — is what you build.