Evaluating an AI Security Platform Through Patterns

You're evaluating AI-powered security platforms for a Cape Verde telecommunications and ISP company facing a SIEM end-of-life and a CEO interested in an "AI-powered security operations platform" he saw demonstrated at a conference. The deliverable is a board-ready recommendation: a pattern-based evaluation framework, three per-platform assessments, a migration architecture, and a 3-year cost comparison.

The discipline skills: assessing portability of detection rules in a proprietary query language; evaluating vendors across data sovereignty, coverage gaps, institutional-knowledge preservation, and total cost; designing a migration architecture that names transition risks honestly; writing a recommendation a board can read and a CEO can argue with.

The AI-direction lesson: all four AI columns express at their deepest register against a tool you have never used. Context strategies, MCP and tool connectivity, the three-layer durability model, and delegation patterns become the evaluation language. Feature lists mislead — they describe what a product calls things, not what it actually supports. The patterns you have built across the track are the questions to ask of any candidate, and the durability of what you build matters more than which product you pick this quarter.

Your Role

You're the consultant Dulce called when her CEO came back from the conference. The framework, the decomposition, the migration architecture, the cost model, the recommendation — all yours. The patterns you have internalised across nineteen projects are the lens you bring.

Scaffolding is thin. No evaluation rubric, no SIEM-migration playbook, no platform-comparison template, no data-sovereignty primer, no portability worksheet for proprietary query languages. No staged colleague review this time — a senior colleague is still on-demand for a second opinion, but they won't volunteer; the evaluation is yours to run.

What's New

Last time, you authored an engagement methodology end-to-end for the Port Authority of Toamasina — formal-memorandum communication, methodology as the central deliverable.

The recommendation is now the deliverable. Not a methodology, not an engagement report — a recommendation the board can decide on, with the framework behind it as the durable artefact.

The patterns turn outward. You spent the track building them by doing the work. This project applies them as the language for evaluating someone else's tool. Engagement memory, MCP, skills, delegation, three-layer durability — the questions you ask of each platform.

Slack throughout. Dulce uses Slack for everything. Direct, pragmatic, occasionally sarcastic about vendor marketing. Tech-aware enough to push back if you propose rip-and-replace without understanding what her team has built. The CEO's enthusiasm has to be addressed honestly without dismissing it.

The hard part: AI will compare platforms by feature matrix if you let it. The pattern vocabulary has to be authored before any platform is evaluated. The data-sovereignty constraint is a hard pass-fail criterion that won't surface unless the framework names it. Five years of institutional knowledge — including three "magic rules" by an engineer who left — has to be reverse-engineered before migration can preserve it.

Tools

• Claude Code, Codex CLI — familiar. Driving the framework, decomposition, migration architecture, cost model, and recommendation draft.
• LogicSearch QL — the proprietary query language the existing SIEM uses. You're assessing portability of five years of rules written in it, not learning to write it from scratch.
• Docker — familiar. The capability inventory and the three platform fixtures run locally.
• Sigma — familiar. Surfaces as the open-standard a portable platform should support at the durable layer.
• MCP — familiar. Surfaces as an evaluation criterion: does each platform support open standards, or lock connectors into proprietary integration?
• Skills (SKILL.md format) — familiar. You author pattern-vocabulary.md as a skill so the framework is reusable for the next tool-change.
• Python, Markdown, YAML, Mermaid, GitHub — for the framework, assessments, migration architecture, cost spreadsheet, recommendation, and Slack record.

No new tool installs versus prior projects.

Materials

• Slack first-contact (first-contact.md) — Dulce's five short messages opening the engagement.
• Pattern vocabulary starting notes (pattern-vocabulary.md) — seed for the SKILL.md-format skill you author in Unit 1. Names the five patterns; the per-pattern questions are yours.
• Regulatory and country context (regulatory-context.md, cape-verde-context.md) — data-sovereignty law, the regulator's upcoming network-traffic-anomaly-detection requirement, currency, linguistic register, submarine-cable interconnect.
• Rede Cabo Verde fixture (rede-cabo-verde-fixture/) — the live SIEM fixture with the rule corpus (five years of LogicSearch QL rules plus the three "magic rules") and a representative log corpus.
• Candidate platforms (candidate-platforms/) — three platforms at distinct points in the design space: VanguardAI (cloud-only, "AI replaces SOC analysts"), OpenSentry (open-standards, Sigma-native), SentinelMesh (hybrid with Sigma at the durable layer and AI-assisted tuning at the volatile layer).
• Engagement memory (engagement-memory/) — the SKILL.md-format pattern-vocabulary template plus templates for the evaluation framework, per-platform assessment, migration architecture, cost model, board-ready recommendation, and handoff checklist. The portability files CLAUDE.md and AGENTS.md sit at the project root, not here.

Everything else — the framework, the decompositions, the migration architecture, the cost comparison, the recommendation, the scope-amendment record, the magic-rules strategy, the Slack record — is yours.