learntodriveai.dev/Web Development/Production Operations: Cost, Observability, and Safe Deployment
Web Development·Project 14·6 units

Production Operations: Cost, Observability, and Safe Deployment.

**Client:** Hannah Bell, Head of Technology and Operations, Altawild Outdoors (Fitzroy, Melbourne, Australia) — a 45-person direct-to-consumer outdoor apparel brand selling merino base layers, packable rain shells, and all-weather accessories. The company started as a 2019 Kickstarter and now ships from a Melbourne flagship to fourteen countries on a Next.js storefront over an Express API gateway with two backend services, RDS Postgres, Redis, and CloudFront in front of S3.

§ Brief

You're taking a live e-commerce platform that has been running for two years and making it operable before its biggest sales quarter. The client is Hannah Bell, Head of Technology and Operations at Altawild Outdoors, a Melbourne outdoor apparel brand shipping to fourteen countries.

The discipline skills: cost investigation on a real AWS bill that doubled with no clear cause, observability used as the primary tool for retrospective incident analysis, deployment strategy calibrated per change rather than per service, SLOs translated into error budgets, and a peak-readiness plan that survives 8-10x baseline load. Every column you've practiced is in play at once. None is climbing. The work is integration, not advancement.

The AI-direction lesson: you treat AI's context and tool connectivity as design decisions. The cost-investigation agent needs the bill and the resource inventory; the incident-triage agent needs the observability stack and the deploy log. Different jobs, different envelopes. You connect a CloudWatch MCP server only for the triage task and disconnect it after — every active connection loads its tool descriptions into every session, and connecting everything is maximum capability and maximum noise. You also decide when a multi-agent pipeline earns its tokens and when one agent is enough. Delegation has a cost.

Your Role

You're a production-operations specialist brought in for one focused engagement. Hannah's eight-engineer team has been heads-down on features; nobody owns the operational layer. Your job is to make the system understood, observable, and changeable without fear, then hand the practice back to her team in a form they can keep using. You don't write a new product. You investigate first, build second, and leave behind artifacts the on-call engineer can operate at 2am.

The register is supervisory. You decide what AI sees, what AI can reach, and which agent does which job. AI runs the queries and drafts the runbooks; you decide which queries matter and what good runbooks look like for this team.

What's New

Last time you decomposed an eight-year-old monolith into five services for a Free Trade Zone distributor in Panama — service boundaries, multi-environment Terraform, a threat model, three AI agents in parallel on a system you were building from scratch.

This project inverts the starting position.

The system already exists. You don't get a PRD. You get a running platform, observability data, deployment history with real incidents, and an AWS bill with a story to tell. The first three units are forensic. AI's instinct will be to skip investigation and propose fixes; holding it in investigation mode until the evidence supports a decision is the directing skill that defines the project.

Context strategy and connectivity become design decisions. What context does each operational task need, what tools should each agent reach, and what should be disconnected when the task is done. This is how the work gets done without drowning in noise.

The deadline is the verification. Black Friday and the Australian summer peak are coming. Your final unit is a load-tested rehearsal of the system at peak with the deploy-freeze policy in force. If it passes, the engagement worked.

Tools

  • The existing Altawild stack — Next.js 14 storefront, Express API gateway, two backend services, PostgreSQL on RDS, Redis on ElastiCache, S3 + CloudFront, third-party shipping APIs, Klaviyo. You modify it; you do not rebuild it.
  • AWS Cost Explorer, Savings Plans, Reserved Instances, Trusted Advisor — new. Cost investigation tooling.
  • CloudWatch, X-Ray, OpenTelemetry, Grafana, Prometheus, Loki, Tempo — continuing. The observability stack is already deployed.
  • Terraform, GitHub Actions, k6 — continuing. CI/CD extended with canary release and a deploy-freeze window.
  • AWS CodeDeploy for canary, synthetic monitoring, and AWS Cost Anomaly Detection or Datadog (you decide) — new.
  • Claude Code with directory-scoped constraints in subdirectory CLAUDE.md files and Codex CLI as MCP server — continuing from P13.
  • GitHub Copilot coding agent — continuing. One async ops change, chosen on cost grounds.
  • AWS CloudWatch MCP server — new. Connected for the incident-triage task only, disconnected after. The unit that uses it walks through setup.

Materials

  • Slack message in #project-altawild — how the project starts. Hannah names the three problems and the November deadline.
  • The Altawild platform — codebase, Terraform, CI/CD, and observability config as the running system has them today.
  • Three months of observability data — covering the May winter-sale slowdown and the steady-state baseline.
  • Twelve months of deployment history — five incidents with their relevant pull requests.
  • AWS billing data — January through April, with the visible doubling.
  • Shipping integration documentation and the 3PL SFTP sync code with a sample failed-sync log.
  • The team's existing CLAUDE.md hierarchy — the root file plus per-area subdirectory CLAUDE.md files, written for build, not for operation. You will critique and refactor.
  • Connectivity decision framework — a reference for the "when NOT to connect" call.

No cost report, no incident retrospective, no peak-readiness checklist, no deploy-freeze policy, no runbook. Those are your deliverables.

Materials

View all →
Dashboards/
Observability Snapshot/
jsondashboards-pre-engagement/error-rates.jsonjsondashboards-pre-engagement/infrastructure-overview.jsonjsondashboards-pre-engagement/request-rates.jsonjsondashboards-pre-engagement/service-health.jsonmdincident-may-14-narrative.mdjsonllogs/logs-slice.jsonljsonmetrics/prometheus-summary.jsonmdREADME.mdotlp-jsonltraces/2026-03/2026-03-14.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-15.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-16.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-17.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-18.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-19.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-20.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-21.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-22.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-23.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-24.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-25.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-26.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-27.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-28.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-29.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-30.otlp-jsonlotlp-jsonltraces/2026-03/2026-03-31.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-01.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-02.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-03.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-04.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-05.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-06.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-07.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-08.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-09.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-10.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-11.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-12.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-13.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-14.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-15.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-16.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-17.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-18.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-19.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-20.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-21.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-22.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-23.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-24.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-25.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-26.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-27.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-28.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-29.otlp-jsonlotlp-jsonltraces/2026-04/2026-04-30.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-01.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-02.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-03.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-04.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-05.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-06.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-07.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-08.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-09.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-10.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-11.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-12.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-13.otlp-jsonlotlp-jsonltraces/2026-05/2026-05-14.otlp-jsonl