You're producing an A/B test readout for an adventure tourism company in La Paz, Bolivia — computing experiment statistics, discovering confounds, and connecting AI to a database via MCP for the first time.
The discipline skills: computing p-values, confidence intervals, and effect sizes from experiment data, defining the success metric precisely before running any test, framing an ambiguous brief into specific analytical questions, and reporting experiment results with honest uncertainty.
The AI-direction lesson: two things shift at once. First, meta-prompting — "I've run a statistical test and I'm not sure the result is trustworthy; help me figure out what could make this test invalid" — uses AI to extend your verification into territory you haven't navigated before. Second, connecting AI to DuckDB via MCP produces a categorical capability shift. Before the connection, AI works from your descriptions and may hallucinate column names. After, it reads the schema directly. The same agent produces different quality output depending on what infrastructure it has access to.
Your Role
You're producing an experiment report: what the A/B test actually shows, what it doesn't show, and what the client should do about it. The brief is deliberately less structured — a situation, not a set of questions. You decide what to investigate, in what order, with what methods.
AI computes the statistics. You verify them against provided targets, catch interpretation errors, and direct the analysis. When AI tells you "there is a 97% probability the new page is more effective," you'll need to know why that sentence is wrong.
What's New
Last time, you integrated four data sources in four formats, designed a metric hierarchy with cascading definitions, and structured findings as a professional argument. You used cross-model review to verify analysis.
This time, three things arrive at once. First: experiment data. A/B tests support causal claims, not just descriptions. The statistical methods overlap with what you've used before, but the inferential stakes are higher. Second: you'll connect AI directly to a database via MCP for the first time. AI reading the schema instead of working from your descriptions changes what it can do. Third: the brief doesn't tell you what question to answer. "Is the new page better?" has multiple valid framings, and the one you choose determines what the test actually tests.
The hard part is not the statistics. It's figuring out what question to ask, discovering the confounds that change the interpretation, and communicating an honest answer when the data tells a more complicated story than Marco expects.
Tools
- Python 3.11+ (via Miniconda, "analytics" environment)
- DuckDB (continuing — now also accessed via MCP)
- DuckDB MCP server (new — first AI tool connection)
- Jupyter Notebook
- pandas
- statsmodels (z-test for proportions)
- scipy.stats (chi-squared and related tests)
- matplotlib / seaborn
- Metabase (via Docker)
- Docker
- Claude Code (plan mode continuing)
- Git / GitHub
Materials
- A/B test dataset — CSV export from Cumbre Adventures' booking platform. About 4,200 rows. Every visitor who saw either page version, what they booked, what they paid, how they arrived.
- Data dictionary — describes the dataset columns and the experiment setup.
- Statistical testing template — reporting format for experiment results: test setup, metric definition, results, confounds, recommendation.
- Verification targets — known-good values for the overall conversion rate test. You compare AI's output against these.
- DuckDB MCP config — configuration file for connecting AI to the database.
- CLAUDE.md — project governance file with client context, work breakdown, and verification targets.