Multi-Metric Experimentation and Multi-Agent Orchestration

You're designing and running ShishuBox's first proper multi-metric experiment for Rafiq Ahmed, Director of E-commerce at a Dhaka educational toy company.

The discipline skills: a primary, secondary set, and guardrails with pre-defined thresholds; correction methods chosen on cost-of-error reasoning (Bonferroni for guardrails, Holm for secondaries); sequential testing where the early-stop value is high; CUPED applied per metric where it earns its keep; the analytical work decomposed across three specialised agents (profiling, analysis, dashboard) with explicit interface artifacts; an audit of the carried-forward AI infrastructure and a path-scoped experiments/CLAUDE.md; a portability test in a second AI tool; an experiment readout that discloses the testing framework; and a framework document Rafiq's product team can use without you in the loop.

The AI-direction lesson sits on three converging surfaces. The family-wise error rate is structural — testing five metrics at alpha 0.05 without correction gives roughly a 23% chance one shows significance by chance, and AI reports each metric in isolation by default. Multi-agent work introduces a new failure mode: each agent produces locally correct output that doesn't compose. The profiling agent flags 12% bot traffic; the analysis agent computes lift on the full dataset; the dashboard shows a "win" that disappears when bots are filtered. AI doesn't propagate findings across agents — you do, through interface artifacts at composition boundaries. And infrastructure that's been "floor that just works" is now load-bearing: a vague skill description misfires, a hook with a buggy script exits zero and gives false confidence, a verbose memory file dilutes the constraint that matters. AI uses what it's given without questioning the quality.

Your Role

You're the analyst Rafiq hired after his team ran one experiment that lifted conversion and doubled return rate. The deliverable is a framework his product team can adopt without you. The experiment is a proof of concept; the documentation is the lasting artifact.

The relationship with AI is operational now. The infrastructure floor — project memory, the profiling skill, the pre-commit hook, MCP scopes, GrowthBook — runs in the background. What's new is that AI directs across three surfaces simultaneously: multi-metric corrections (where AI applies the math but doesn't surface the cost-of-error reasoning), multi-agent orchestration (where you are the keeper of global coherence), and infrastructure quality at scale. Verification techniques get calibrated by layer rather than run uniformly: inline at the agent-output level, cross-checks at composition boundaries, meta-prompted design at the workflow boundary.

What's New

Last time you stood up BigQuery and dbt for Aisha at Kapra Market and moved metric definitions out of prose and into version-controlled YAML, with the cross-system check landing at the warehouse and at the compiled SQL.

This time the cross-system check moves between agents. The bot-traffic finding from a profiling agent must reach an analysis agent and end up reflected in the dashboard. AI's first composition produces a "winning" experiment that loses when bots are filtered, and you catch it through the interface chain — profiling-report.md to analysis-report.md to the dashboard's source comment — not through agent-by-agent review.

Rafiq is a different kind of audience: long, well-organised emails and clarifying follow-ups, but the framing work for what experiment runs first, which guardrails matter for ShishuBox specifically, and where sequential testing fits is still yours. Three things are genuinely new: multi-metric experiment design with corrections, guardrails, CUPED, and sequential testing; the three-agent decomposition with explicit interface artifacts; and an infrastructure quality audit followed by a deliberate portability test in Codex CLI, where a skill with a Claude-specific path convention silently fails and AGENTS.md is the gap you fix.

The hard part is the moment AI's first multi-metric report shows "secondary metric B improved (p=0.03)" without the multiplicity context, and you have to restore the testing framework — uncorrected and corrected p-values, correction-method reasoning, guardrail-threshold conditions — before Rafiq's product team adopts the framework as written.

Tools

• Established and carried forward: Claude Code, Codex CLI (used here for the portability test), DuckDB and DuckDB MCP, BigQuery and BigQuery MCP, dbt Core (read-only — the experiment configuration consumes its definitions), GrowthBook (multi-metric, guardrail, sequential-testing, and CUPED features exercised here), Python with pandas/scipy/statsmodels, Jupyter, Git and GitHub with pre-commit, the root CLAUDE.md, the profile-dataset skill (audited and rewritten here), the pre-commit hook (extended to govern experiments/).
• Path-scoped project memory — new; experiments/CLAUDE.md authored alongside the root, scoped to the experiments directory.
• Multi-agent decomposition pattern — new; three specialised agents with explicit interface artifacts at composition boundaries.
• Per-agent context briefs — new; short, scoped context files per agent rather than one shared context.
• Async delegation via GitHub issue + PR — new; used for the dashboard agent, in contrast to the conversational orchestration used for the profiling-and-analysis pair.
• AGENTS.md — new; authored after the Codex portability test surfaces the gap.

Materials

• Rafiq's first-contact email — long, well-organised paragraphs covering background, history, current state, and what he's asking for. Specifics on thresholds, guardrails, and statistical framework are still yours.
• Synthetic ShishuBox experiment dataset in BigQuery — six weeks of pre-experiment data and four weeks of experiment-window data across traffic, experiment_assignments, orders, subscriptions, returns, and nps_responses. About 1.2M traffic events, with embedded bot-traffic (~12%), age-group cookie segmentation, and the subscription-vs-one-time channel split.
• Pre-experiment historical export — the previous quarter's per-user AOV, sessions, and conversion rate, for the CUPED covariate.
• Carried-forward dbt project (P15) — read-only; loaded against ShishuBox's BigQuery dataset, with definitions for conversion_rate, signup_rate, average_order_value, return_rate, nps_score, and subscription_retention.
• Carried-forward GrowthBook instance (P14) — configured against ShishuBox's data sources, no experiments yet defined.
• Starter experiments/ directory — experiment-configuration template, empty per-agent brief files, and an empty experiments/CLAUDE.md placeholder.
• Starter CLAUDE.md — context and tech stack populated. Correction policy, guardrail-threshold table, CUPED log, infrastructure audit, portability results, and framework documentation are placeholders you fill in.
• Reference primers — multi-metric corrections, guardrail-threshold reasoning, CUPED applicability, sequential testing, and multi-agent decomposition patterns.
• Established profile-dataset skill and .pre-commit-config.yaml — carried forward and audited; the skill description gets rewritten and the hook regex gets extended.

Download the materials zip (https://learntodriveai.dev/materials/analytics/p-16/materials.zip) and unzip it to ~/dev/analytics/p-16.