learntodriveai.dev/Cloud Engineering/Cascading-Incident Response and Communication Under Pressure
Cloud Engineering·Project 16·7 units

Cascading-Incident Response and Communication Under Pressure

**Track:** Cloud Engineering

§ Brief

You're designing the incident-response framework for Erik Lindqvist, the SRE team lead at NordPay, a Swedish neobank serving three million customers across the Nordics, where a recent 3-hour cascade fired 47 alerts and the team could not tell which alert was the actual problem.

The discipline skills: a runtime dependency graph reconstructed from a distributed trace; a symptom-vs-cause separation procedure read against the trace, not the alert list; a blast-radius gate that requires a one-line worst-case statement before any proposed fix is applied; a three-role split — Incident Commander, Communicator, Investigator — with team-size and duration triggers; stakeholder-update templates in three registers; and a post-incident review that produces architectural recommendations.

The AI-direction lesson is that AI cannot tell cause from symptom in a cascade. Asked why a service is timing out, it lists ten plausible causes and treats every one as evidence. Asked how to clear a saturated connection pool, it suggests restarting the database without naming the in-flight transactions lost. Asked to draft a status update, it writes the same sentence for engineers and customer success. The chain of causation is in the trace; the worst case is in the gate; the audience is in the register.

Your Role

Erik knows distributed systems. He has read the SRE book and understands cascading failures as concepts. He is direct without being abrupt — the analytical kind, takes a moment before speaking, appreciates being challenged back. What failed during The Cascade was the ability to reason under pressure and communicate to stakeholders without taking the team off the trace. The first contact is a four-sentence Slack message. There is no methodology document, no template, no procedural reference — the engagement's deliverables are your design.

You direct Claude through the dependency-graph reconstruction, the framework drafting, the policy-as-code that enforces the blast-radius gate, the simulated-incident walkthrough, and the stakeholder-update templates. Marco Reyes, the staff SRE from the previous engagement, appears three times to ratify edges from his on-call experience — at the symptom-vs-cause framework, the blast-radius gate, and the post-incident review. He does not propose the framework's structure. You do.

What's New

Last time at Conecta Móvil you classified toil against a six-criterion definition, deployed 15 services through ArgoCD with sync-wave ordering, instrumented OpenTelemetry across metrics, logs, and traces, and designed three MCP permission scopes. You leave with the dependency-graph-as-substrate habit, the three-pillars vocabulary, and a CLAUDE.md that grows with the work.

This time the operational stream is the substance under the hardest conditions. Incidents stop being a single alert with a runbook and become a cascade where five alerts fire and only one is the cause. Plan review stops being a terraform plan diff and becomes a worst-case statement on every proposed fix. Communication stops being a weekly status update and becomes three audiences at once. The engagement opens with a Slack message — you design the deliverables.

The hard part: AI's symptom list is plausible without being causal. Restarting the database is fast without being safe. The status update is precise for engineers without being readable for customers. Every default is reasonable on its own and dangerous in a cascade.

Tools

  • CLAUDE.md — extended with two new Conventions: the blast-radius gate and the incident-roles split.
  • OPA + conftest — extended with a new policy that requires a non-empty worst_case field on every proposed fix before approval.
  • scripts/verify.sh — extended with the dependency-graph completeness dimension.
  • AWS Console + CLI, Cost Explorer, AWS Budgets — carrying forward.
  • Terraform — used here for a budget alarm and a small CloudWatch-alarms stack for the simulated incident.
  • AWS MCP server and Datadog MCP server — both read-only, reused for the walkthrough.
  • CloudWatch, CloudWatch Logs Insights, X-Ray, Datadog — the four signal stores you read during the cascade.
  • Claude Code — carrying forward.

Materials

In the project zip:

  • Erik's first-contact Slack message (erik-first-contact-slack.md) — four sentences, the SRE register.
  • NordPay service list (nordpay-service-list.csv) — seven rows for the services involved in The Cascade. The runtime dependency graph is not in the file.
  • CloudWatch metrics export (cascade-metrics-export.json) — 47 alert events from the 3-hour incident. Two are false positives.
  • CloudWatch Logs export (cascade-logs-export.json) — application logs from the seven services across the incident window.
  • X-Ray trace export (cascade-trace-export.json) — distributed traces. The chain of causation is here.
  • Incident-history note (incident-history-note.md) — the public-facing summary Erik wrote for his executive after The Cascade.
  • Dependency-graph script starter (scripts/check-dependency-graph.py) — a TODO body. You write the implementation.
  • CLAUDE.md — engagement facts in place and two TODO markers for the new Conventions.

You produce the runtime dependency graph, the symptom-vs-cause procedure, the blast-radius gate as Convention and policy, the simulated-incident walkthrough, the incident-roles framework, the stakeholder-update templates, the post-incident-review template, and the handover.