Ingrid Halvorsen runs the data team at Nordfjord Aqua AS, a salmon aquaculture company on the Nordland coast in Norway. Twelve farm sites, three families of sources on each — sensors, feeding systems, fish-health records — and no central view yet. She needs an ingestion architecture that decides which sources stream, which use change capture, and which run as daily batch.
The discipline skills: designing the ingestion architecture as a system of per-source decisions — freshness, source-system load, cost, operational complexity. Streaming where it earns its cost. CDC without degrading the source database. Batch where batch is the right answer. Schema-registry compatibility — BACKWARD versus FULL — chosen per event family. Governance at landing, not at consumption. An architecture document a marine biologist can read and a data engineer can run.
The AI-direction lesson: AI given "design the ingestion layer" produces a uniform architecture — every source streamed, or every source batched. The judgment is per source and has to be made before AI writes any code. You decompose first — source inventory, freshness per source, source-system load, cost ceiling — and direct AI against named constraints. A daily batch that meets the consumer's actual need is simpler, cheaper, and more reliable than a streaming pipeline delivering freshness nobody uses; knowing when streaming is overkill is part of the work.
Your Role
You are the ingestion architect designing the source landscape from scratch. You decide the patterns, write the architecture document, build representative pipelines that prove each pattern, and hand back something Ingrid can sign off on and her team can run for years.
The scaffolding is thinner than anything you have seen. No architecture template, no per-source decision tree, no spec hand-off. Ingrid hands over a problem and a constraint set; everything else is yours — the document's structure, the source inventory, the per-source decisions, the verification plan, the per-source-family agent contexts. Your existing repository — dbt, BigQuery, Dagster, Kafka, Debezium, the path-scoped CLAUDE.md, your MCP setup, your hooks — carries forward. You extend it; you do not start over.
The AI relationship sits one register higher. You decompose before AI writes anything, evaluate per-source recommendations as a system, and hold global coherence across what three agents produce. JT Thompson returns as senior colleague — terse, infrastructure-savvy, the right pair of eyes for WAL-load and cost-shape questions before they ship.
What's New
Last project you compared two warehouses and two orchestrators head-to-head for Diego's poultry cooperative and shipped a recommendation grounded in cost.
This project the comparison axis disappears. Nordfjord is committed to BigQuery and Dagster — the question is which ingestion pattern per source. Genuinely new: the schema registry as a first-class deliverable. Per-source-family agents — sensor, feeding, fish-health — each with selective MCP connectivity and the discipline of refusing the connections they do not need. Discovery through Slack DM rather than forwarded email — Ingrid is in-thread throughout, reserved and precise, sharp on fish biology. And the architecture document itself, written from scratch.
The hard part: per-source decisions are silent failures waiting to happen. Stream a sensor that updates every 15 minutes and you burn cellular bandwidth for freshness nobody uses. Run CDC on the production feeding-system database without coordinating with the DBA and the database starts swapping. Add a required Avro field without a default and every existing consumer breaks. Take "we need real-time" at face value from a team that opens the dashboard once a shift and you over-engineer something costing ten times what daily batch would. The architecture has to defend against all of it before any code runs.
Tools
- dbt, BigQuery, Dagster, Kafka, Debezium, Soda Core, DuckDB, PostgreSQL, MinIO, Docker Compose, GitHub Actions, Claude Code, Codex CLI, MCP — all carrying forward from prior projects
- Confluent Schema Registry (or open Apicurio equivalent) — new, for Avro schema evolution. Local Compose service alongside your existing Kafka stack; the unit that introduces it walks through setup and compatibility rules.
- A Nordfjord source generator — new, produces the three source families across 12 sites. Run once at project start.
Materials
You'll receive:
- A project governance file (
CLAUDE.md) — Nordfjord client context, Ingrid's constraint set, verification targets, commit convention. - The Nordfjord source generator — 12 sites of sensor streams, feeding-system change events, and fish-health records.
- A schema registry Compose overlay — stands the registry up alongside your existing Kafka stack.
- A per-site connectivity profile sheet — 12 sites tagged fiber, 4G cellular, or satellite.
- Ingrid's introductory Slack DM — in the chat at project start.
Deliberately not provided: any template for the architecture document, source inventory, per-source decisions, or per-source-family agent contexts. Your existing repository carries forward as the working platform.