The supermarket loyalty analysis is back for a second pass. You deepen the causal validation -- sensitivity analysis across specifications, refutation interpretation, and a board-ready analysis that survives scrutiny from quantitative stakeholders.
The discipline skills: sensitivity analysis across propensity score specifications, interpreting refutation tests beyond "passed" or "failed," configuring multiple MCP connections with deliberate permissions, designing a custom statistics-review agent, and running cross-tool verification with Codex CLI.
The AI-direction lesson: you design a statistics-review agent that evaluates the methodology independently -- and then you must review analytical work you did not watch happen. When a delegated agent returns its findings, you evaluate judgments made by something you designed but did not direct step by step. The skill is reviewing output without having seen the intermediate reasoning. You also learn that delegation scope matters: an agent with access to your methodology memo may rubber-stamp instead of independently evaluating. The same agent without the memo evaluates honestly.
Your Role
You deliver confidence in the causal estimate -- or honest caveats about where confidence breaks down. This time, you manage multiple AI interactions simultaneously, configure permission boundaries, and deploy a custom agent for independent review. The analytical territory is familiar. The infrastructure is not.
What's New
Last time, you built the entire causal analysis from scratch -- the DAG, the propensity score model, the refutation tests, the honest communication of 35% vs 8%. You worked with one AI tool, one MCP connection, and turn-by-turn directing.
This time, the analytical question is settled. The new terrain is AI infrastructure: multiple MCP connections with deliberate permission design, a custom statistics-review agent with scope boundaries you define, and cross-tool verification using Codex CLI alongside Claude Code. You also go deeper on validation -- sensitivity analysis across specifications and refutation interpretation beyond "the test passed."
The hard part is reviewing work you did not see being created. When the statistics-review agent returns its findings, you must evaluate analytical judgments made by an agent you designed but did not direct step by step.
Tools
- Python 3.11+ via your conda "ds" environment
- Jupyter Notebook for the analysis
- Jupyter MCP server -- connects AI to notebook execution (new)
- pandas and Polars for data handling
- DoWhy for causal inference
- statsmodels and scipy for statistical analysis
- matplotlib / seaborn for visualization
- DuckDB and DuckDB MCP server (carry-forward)
- Codex CLI -- a second AI coding agent for cross-tool verification (new)
- Claude Code as the primary AI you direct
- Git / GitHub for version control
Materials
You receive:
- The same transaction dataset from last time -- 14 months of loyalty program data across all 35 stores
- A data dictionary describing the columns
- A validation note from the client's finance team (reviewed in Unit 3)
- No methodology templates, no agent configuration templates, no analysis guides