Evaluating a Tool Landscape Change

Armen Hovhannisyan, CTO at PayNow Armenia, has 14 pipeline configs running on a data orchestrator that just announced a breaking change in its next major version. Half won't run after the 90-day EOL, and the finance reconciliation pipeline runs daily for the Central Bank of Armenia and cannot go down.

A note on the scenario: the "Dagster 2.0 / 90-day EOL" tool-landscape change is a constructed scenario for project pedagogy. Dagster's actual release line in 2026 is still 1.x; the orchestrator-changelog.md reference in materials/ is a fictional artefact built in the shape of a real vendor changelog so this engagement has a concrete breaking-change to evaluate against. The transferable skills — three-layer durability vocabulary, migrate/stay/switch evaluation, parity-check encoding — carry across orchestrators and across real release cycles.

The discipline skills: a three-layer durability framework written down explicitly — open standards, cross-platform patterns, tool-specific features; a 14-config inventory classified against it; a migrate / stay / switch evaluation memo in Armen's terse register; a staged migration plan that respects the daily-reporting cadence; a parity check on the finance reconciliation cutover encoded as a GitHub Actions hook the merge cannot bypass.

The AI-direction lesson: the durability vocabulary you have accumulated implicitly across the track is a transferable framework — writing it down is what makes your experience portable. AI defaults to "migrate to the new version" with an effort estimate; you hold the three-option discipline and require AI to evaluate each option against the framework first. AI proposes vendor comparison from changelogs; you ground the evaluation in what you have built. AI under-prices the volatile tool-specific layer. AI writes a clean migration plan that ignores the production calendar.

Your Role

You are the architect. Armen sends a Slack message. You reply in his register, open a targeted-discovery thread, read the 14 configs directly, write the three-layer framework as the explicit deliverable, classify every config against it, evaluate the new version and two alternative orchestrators, build the recommendation, stage the migration around the off-cycle window, encode the parity check as a hook, run the cutover, and hand PayNow a portable engagement bundle.

The scaffolding is thin. A Slack message and a 90-day vendor-imposed deadline are the entire brief. No framework template, no inventory template, no migration-plan template, no parity-check template. Everything is your design.

The AI relationship is supervisory-to-autonomous inside a framework you author. AI does not propose the framework, the classification rules, the recommendation, or the parity logic. AI reads each config, drafts the per-config rewrite against the new version's API, and runs the parity query — inside the structures you designed. The cutover hook is the most autonomous expression: the parity check runs in GitHub Actions on the cutover commit, and the merge cannot complete unless row count, total amount, and per-merchant settlement match between the old and new versions. AI cannot override the hook.

What's New

Last project you evaluated an eleven-pipeline portfolio for Marlene Castellanos at the Belize Tourism Board — per-pipeline build-vs-buy, three-year cost forecast, transition planning around the cruise-season reporting calendar, a board-defensible recommendation. The framework-before-execution discipline carries forward.

No colleague reviews the work — Armen is the only client check-in, and his bandwidth is measured in minutes. (Tigran Marutyan, a data-reliability engineer, is reachable on demand for a narrow sanity check on whether the finance-reconciliation cutover is protected by a parity gate the merge cannot bypass; he volunteers nothing and writes no part of the migration.) Genuinely new: a vendor-imposed EOL deadline (every previous deadline was client-set); the three-layer durability framework written down as an explicit deliverable PayNow inherits; evaluating alternative orchestrators by applying a pattern vocabulary, not a feature-matrix; a staged migration executed end-to-end rather than recommended on paper; a parity check encoded as infrastructure the cutover cannot bypass; Armen's terse Slack register.

The hard part: holding the three-option discipline when Armen says "I'm thinking migrate." The undocumented /internal/payout-states endpoint in two configs is being removed entirely. The finance reconciliation pipeline relies on an execution order coded by a contractor who left six months ago; the new version's scheduler may reorder the steps and the settlement totals will be wrong but the pipeline will still run. The cutover window is overnight; the regulator reads the daily report at 6 AM; a row-count mismatch is a regulator-flag event.

Tools

• Claude Code, Dagster (the current orchestrator, 1.x → 2.x as the canonical migrate target), dbt, BigQuery, Soda Core, GitHub Actions, Git + GitHub, CLAUDE.md + AGENTS.md as portable engagement memory — all carry-forward.
• Alternative orchestrators evaluated against the framework but not implemented: Airflow (the AI-popularity-default trap) and Prefect (the per-pattern-fit candidate).
• Managed-tool references named in the alternatives evaluation but not implemented: a managed CDC service (Fivetran or Airbyte) as a partial-switch option for the merchant transaction stream; a managed observability layer (Monte Carlo or SYNQ) if Armen's scope-creep beat about better monitoring is accepted.

No new tools. What is distinctive is the use: the three-layer durability framework as the explicit evaluation contract, and a parity check encoded as a GitHub Actions hook with no AI-override path.

Materials

You'll receive:

• A project governance file (CLAUDE.md) — the engagement repository's session context: scope, the three-layer durability vocabulary, the constraint-flow sequence, verification targets, recommendation-register rules, commit convention.
• The cross-tool mirror (AGENTS.md) — the same content reformatted as the open-standard project-memory file, so the engagement can be re-opened in any AI tool.
• Armen's opening Slack message (armen-slack.md) — channel #urgent-data, 11:23 AM, three short paragraphs. The register you reply in.
• PayNow operational picture (paynow-context.md) — the 40-person Yerevan fintech, the 50,000-transactions-a-day scale, the Central Bank of Armenia reporting commitment, the 14-pipeline portfolio named at the function level, the team and contractor history, the operational constraints.
• The 14 pipeline configs (paynow-configs/) — representative Dagster 1.x files you read directly against the framework. Realistic in shape so each layer of the durability framework is legible in the code.
• The orchestrator changelog (orchestrator-changelog.md) — the breaking-changes reference for the new major version: decorator and config-API renames, sensor-signature changes, the scheduler-behaviour change for chained tasks, the removed undocumented endpoint, the compatibility shim.
• The alternatives reference (orchestrator-alternatives-reference.md) — Airflow and Prefect profiled at evaluation depth, plus a closing per-pattern-fit summary. No recommendation; the decision rules are yours.

Deliberately not provided: the durability framework, the classification criteria, the migrate / stay / switch decision rules, the migration-staging rules, the parity-check specification, the recommendation register. The BTB engagement workspace from last project does not carry forward — fresh project root for PayNow.