Environment Audit, Observability Architecture, and the Tooling Question

You're directing an audit and observability engagement for Elif Demir, Director of Research Computing at Istanbul Technical University — a Turkish university running cloud infrastructure for 12 research groups across four years of un-governed growth.

The discipline skills: auditing an environment with partial IaC coverage across multiple accounts; classifying each resource as import, replace, or decommission with behavioral evidence behind the call; writing a dependency-aware import order and executing it level by level; evaluating OpenTofu against the imported Terraform state and writing the recommendation for a non-technical governance committee; designing an observability architecture that names monitoring and observability as separate layers; writing SLOs with error budgets calibrated for research workloads; and designing for handover so a successor can operate the system without your explanation.

The AI-direction lesson is that a methodology is the only thing that survives an engagement at this scale. AI produces inventories, not audits — flat lists without the judgment dimension. It proposes imports in alphabetical order, which violates the dependency graph. It generates monitoring when asked for observability because monitoring is what its training data is dense on. It does not raise residency constraints unprompted, and omits the handover dimension. You design the methodology AI applies, and catch the categorical confusion when an "observability architecture" arrives as a CloudWatch dashboard.

Your Role

You own the audit methodology, the import dependency graph, the OpenTofu evaluation, the observability architecture, and the handover README. The deliverables are mostly markdown — the Terraform footprint is intentionally small. The substance is the documents and the judgment behind them.

Communication switches back to asynchronous email at academic-rigour cadence. Elif replies within a business day with structured numbered paragraphs that mirror your question, and asks "what's your methodology?" before approving anything. Priya Subramanian appears twice as edge-ratifier — once on import methodology, once on a stale-state-file warning — but does not propose the design. You do.

What's New

Last time at FSS you designed AI infrastructure — a security-audit skill, two fail-closed hooks, a verification architecture for a three-agent canary. The hooks and skill carry forward here as substrate; they are not the substance.

This time the discipline columns return to centre stage. The work spans 12 research-group accounts, not one engagement's deliverables. The audit is not an inventory — every row needs evidence (CloudWatch invocations, access logs, a conversation with the group that owns the resource) and a classification rationale. Communication switches from in-session meeting notes back to formal email, and you also write a proposal for a non-technical IT governance committee.

The hard part: AI fails in categorical ways at this altitude. A monitoring system labelled "observability" passes its own validation while missing the architectural distinction the engagement rests on. An import list in alphabetical order produces post-import plans full of phantom diffs AI then "fixes" by modifying parent resources. A residency violation sits silently in a clean-looking inventory. Catching these requires understanding what the deliverable is supposed to do, not whether the output looks well-formed.

Tools

• Terraform — reused throughout the audit for plan, import, state, and the representative import slice.
• OpenTofu — new. Installed in Unit 5, stood up against the imported Terraform state for the evaluation.
• AWS Console and CLI — heavy use during the discovery sweep across 12 research-group account aliases.
• AWS MCP server — reused from P12, read-only, with a new namespace exclusion under ITU's GDPR-residency Non-Negotiable.
• CloudWatch and CloudWatch Logs Insights — the behavioral-data source for the audit's in-use determination, and the observability platform for Unit 6.
• CLAUDE.md — extended with three new entries: the GDPR-residency Non-Negotiable and two Conventions (audit methodology and observability handover discipline).
• scripts/verify.sh — extended with one new dimension (check-audit-completeness.sh) asserting a zero-change plan after import.
• Mermaid and Markdown — the dependency graph, the inventory, the architecture, the OpenTofu proposal, and the handover README.

Materials

In the project zip:

• Elif's kickoff email (elif-kickoff-email.md) — the four-paragraph first contact, no calibrations.
• ITU institutional context (itu-institutional-context.md) — Elif's one-page reference: the 12 research groups, the account aliases, the budget cycle, the governance committee composition.
• GDPR-compliance checklist (itu-gdpr-checklist.md) — a half-page draft from ITU's data-protection office. The mapping to AWS-specific actions is not in the file.
• Firefly 2025 IaC adoption excerpt (firefly-iac-adoption-2025-excerpt.md) — the headline Terraform/OpenTofu statistics. No evaluation framework — you write one.
• A sample EU-funded dataset (sample-horizon-europe-dataset/) — the climate-modelling slice the GDPR remediation migrates from us-east-1 to eu-central-1.
• Terraform starters (terraform/budget/, terraform/import-target/, terraform/observability-hpc-submission-api/) — the budget-alarm module, the import-target shell, and the observability-module skeleton.
• Script starters in scripts/ — TODO-body shells for the audit-completeness check and import/observability preflights.
• CLAUDE.md — engagement governance with three TODO markers for the new Non-Negotiable and two Conventions.
• A framework/ directory — empty, where the methodology, inventory, architecture, OpenTofu proposal, and handover README will live.