Experimentation Platform: Setup and Manual Verification.

You're standing up GrowthBook for Tomáš Dvořák, Head of Product at TaskForge — a Prague SaaS project management platform — and running their first real experiment on it.

The discipline skills: configuring an experimentation platform end-to-end (data source, metric library, experiment, variants), defining metrics tool-agnostically and again in the platform's format, running a manual Python analysis to mirror the platform's, reconciling the two, and writing a one-page readout plus a repeatable-process runbook.

The AI-direction lesson: an experimentation platform is an automation layer, not an authority. GrowthBook computes the same statistics you would compute in Python, but it applies its own defaults — significance level, frequentist or Bayesian, sequential testing on or off — and AI tends to accept those defaults silently. The manual Python analysis is the audit. When the two numbers agree, the platform earns trust on that metric. When they diverge, the divergence usually points to a configuration choice the platform made on your behalf. You also decide per-metric how much cross-check is warranted.

Your Role

You're the analyst standing up the experimentation program. Two artifacts have to land for two different audiences: a one-page readout for Tomáš in his own register, and a runbook the TaskForge product managers can follow without an analyst in the loop every time.

The relationship with AI deepens rather than shifts. The infrastructure floor — project memory, the profiling skill, the pre-commit hook, MCP scopes, the second AI tool — is established and runs in the background. What's new is that you direct AI across two computation paths whose outputs must agree: AI configures GrowthBook on one side and writes the manual Python analysis on the other. The divergence between them is yours to surface.

What's New

Last time, you built an investor-grade Tableau dashboard for a Montevideo CFO and verified it against a Metabase reference — two BI platforms, both transparent, the same metric defined once and implemented twice.

This time, the cross-system check stretches further. One side is the platform — GrowthBook's statistical engine, opaque, configured through a UI and an SDK. The other side is the manual Python analysis you write yourself, transparent end-to-end. AI silently picks the platform's defaults on one side and keeps its established failure patterns (wrong test, missing correction, overclaim of certainty) on the other.

The audience is also new. Tomáš writes like a developer in Slack — short, casual, "ship it." The readout has to match his register. The runbook is for his product managers, not analysts.

The hard part is not the analysis. It's the moment the platform reports one number and the notebook reports another, and you have to investigate which side is right before you take a verdict to Tomáš.

Tools

• Claude Code with the DuckDB MCP connection — established
• Codex CLI — established, available but used sparingly
• DuckDB — established
• Python (pandas, numpy, scipy, statsmodels) and Jupyter — established
• Git and GitHub with the pre-commit framework — established (the hook regex extends to govern notebooks and experiment artifacts)
• CLAUDE.md and the profile-dataset skill — carried forward
• GrowthBook — new; the experimentation platform. The unit that uses it walks through the choice between cloud free tier and self-hosted Docker.
• GrowthBook SDK + API key — new; the integration pattern. Not MCP. A community-built GrowthBook MCP server exists; whether to adopt it is a deliberate decision.

Materials

• First-contact Slack message from Tomáš — three lines, the entire entry artifact. The framing work happens in conversation.
• TaskForge event-tracking dataset — users.csv (about 18,000 user metadata rows) and events.csv (about 250,000 event rows: exposures, activations, paid upgrades, engagement). The experiment-eligible subset is free-tier signups in the experiment window, about 4,500-5,500 users.
• Data dictionary — column schema, ranges, null policy, joins, and the structural attributes the profiling skill cross-checks.
• Starter CLAUDE.md — project context, tech stack, the established skill, and the established hook are populated. Metric definitions, the GrowthBook trust evaluation, the configuration decisions, and the reconciliation rules are placeholders you fill in across the project.
• Reference material — a GrowthBook orientation primer, a GrowthBook trust-profile note, and a one-page reference on practical-significance questions.
• Optional docker-compose.yml — the self-hosted GrowthBook stack, for the route that doesn't use the cloud free tier.
• Established profile-dataset skill and .pre-commit-config.yaml — copied in. You invoke and extend them; you do not re-author them.

A senior colleague — Petra Nováková, an independent Prague-based experimentation consultant — is available on a separate Slack-style chat. She is terse and busy. Worth pinging when a stake-versus-effort question is worth a second opinion; not someone who'll do the work for you.

Download the materials zip (https://learntodriveai.dev/materials/analytics/p-14/materials.zip) and unzip it to ~/dev/analytics/p-14.