Experiment Lifecycle Ownership and Metric Governance

You're designing a dynamic-pricing experiment end-to-end for Ryan McKenzie, the General Manager of an independent Queensland car-rental company deciding whether to move off fixed daily rates.

The discipline skills: a business-first chain answered before any statistics — what change, why, what counts as success, what could go wrong, how long the business can wait, and what action follows each outcome; one-tailed vs two-tailed selection per metric, recorded as a claim about expected effect direction; significance level chosen per metric against the cost of false positives and false negatives; leading and lagging indicators split deliberately, with the lagging set tied to roll-back triggers that run for weeks past the test conclusion; a first-draft metric governance framework for Ryan's small team; a documented connectivity architecture across the five phases of the lifecycle; and three audience-calibrated reports from one experiment.

The AI-direction lesson sits on three converging surfaces. AI skips the business-first chain and goes straight to statistical design — two-tailed by default without naming the choice, alpha = 0.05 by convention without surfacing cost-of-error reasoning. AI also generates governance frameworks that are structurally complete but organisationally naive — a Metrics Committee with quarterly reviews for a 55-person operator with no committee culture. And the lifecycle has phases — discover, design, run, analyse, report — each with its own tool set, context, and AI involvement. You scaffold the chain, replace the committee with a structure that fits the actual team, and stay the keeper of coherence across phase boundaries: the alpha you chose during design has to reach the analysis without drift, and the analysis findings have to land consistently in three reports without diverging.

Your Role

You're the analyst Ryan reaches out to because he wants the experiment run properly, not as a switch he flips. The deliverables are an experiment that answers his business question, a governance framework his small team can use without you, a connectivity architecture another analyst could follow, and three reports for three audiences.

The relationship with AI is operational across the lifecycle. The infrastructure floor — project memory, AGENTS.md, the profiling skill, the pre-commit hook, MCP scopes for DuckDB and BigQuery, dbt definitions, GrowthBook — runs in the background. What's new is that your judgment now spans phases rather than agents: per-phase context scoping, per-phase tool sets with explicit permissions, and verification at phase boundaries. AI runs as an MCP server in one phase — Claude Code exposing a single scoped review capability that another tool connects to — and you size the permission scope.

What's New

Last time you decomposed a multi-metric experiment for Rafiq across three specialised agents, with explicit interface artifacts at the composition boundaries and a portability test that surfaced the AGENTS.md gap.

This time the decomposition is across the lifecycle, not across agents. The first contact is a two-minute voicemail; the discovery instrument is a recorded call back to Ryan, and the transcript becomes a working material. The brief is symptoms only — everything past it is yours to design.

Three things are genuinely new. Full experiment design end-to-end via the business-first chain, with the test-type and alpha selections recorded per metric and a leading/lagging framework that extends impact measurement past the test conclusion. A first-draft governance framework — single named owner, change process, per-experiment review trigger — written for a 55-person team where AI's first draft proposes a Metrics Committee. A phase-scoped connectivity architecture written into a documented artifact, with one phase exercising the dual nature of MCP — AI tool as server, not just client.

The hard part is the moment AI's audience-report drafts have stopped agreeing with each other. You restore the conditions per audience and check that the alpha values, the exclusions, and the headline finding still match across all three.

Tools

• Established and carried forward: Claude Code, Codex CLI, DuckDB and BigQuery (with their MCP servers), dbt Core (read-only), GrowthBook (multi-metric, guardrails, sequential testing, CUPED), Metabase, Python with pandas/scipy/statsmodels, Jupyter, Git and GitHub with pre-commit, the root and experiments/ CLAUDE.md and AGENTS.md, the profile-dataset skill.
• Phase-scoped connectivity architecture — new; an infrastructure/connectivity.md artifact laying out the five lifecycle phases with their tool sets, permission scopes, and the rationale for connections deliberately absent.
• AI tool as MCP server — new; Claude Code configured to expose a single scoped review capability, with Codex CLI connecting to it for one round-trip during the design phase.
• Voicemail and recorded discovery call — new artifact register; the first contact is voice, and the call-back transcript is a Unit 1 material.

Materials

• Ryan's voicemail transcript — the entry artifact. Sparse, casual, two minutes. Test design, metric thresholds, and the exclusions that matter are yours to surface in the call back.
• Recorded discovery call setup guide — a short reference on tooling, transcription, and the structure of targeted questions.
• Synthetic Coral Coast booking dataset in BigQuery — three years of pre-experiment history (~150,000 rentals across four locations and seven vehicle categories) plus six weeks of experiment-window data. Tables: bookings, vehicles, assignments, pricing_history, customer_reviews, nps_responses, competitor_rates. A pre-experiment historical export is included for the CUPED covariate.
• Carried-forward dbt project and GrowthBook instance — both loaded against Coral Coast's data sources, with the six fleet-pricing metric definitions in dbt.
• Starter directories — experiments/ (path-scoped memory carrying forward from P16), infrastructure/connectivity.md (empty template with the five phase headers), governance/framework-draft.md (empty template with metric-owner, change-process, and AI-naivety-correction headers).
• Reference primers at reference/ — the business-first chain, test-type-and-alpha, leading-vs-lagging, multi-audience reporting, phase-scoped connectivity, governance as organisational work.
• Project governance file (CLAUDE.md) — context and tech stack populated; the discovery notes, chain answers, experiment design, connectivity summary, governance summary, verification extensions, and three audience reports are placeholders you fill in. The profile-dataset skill and .pre-commit-config.yaml carry forward from P16.

Download the materials zip (https://learntodriveai.dev/materials/analytics/p-17/materials.zip) and unzip it to ~/dev/analytics/p-17.