Deployment Order — Standing Up the Estate

The order to apply the Terraform (OpenTofu) roots when building eSHARS infrastructure, and the post-apply steps that make an environment actually work. Get the order wrong and an apply fails (or silently skips wiring), because each root reads things the earlier roots create.

For why configuration and secrets flow the way they do, read Configuration & Secrets Architecture first. For the detailed per-environment procedure (secret seeding, SQL logins, the readiness gate), this page links into the Workload-env Stand-up runbook.

The chain is automated — workflows are the primary path

This entire order is codified as the Infra: 1–6 GitHub Actions workflows in eshars-ops, chained by the Infra: 0 - Stand-up orchestrator and fronted by a Port self-service action. A from-zero tier stand-up is one dispatch, not a shell session. The manual just commands below remain valid (and are what the workflows run under the hood) — use them for single-root iteration; use the workflows for stand-ups. See Stand-up Workflows.

The short version

Building a brand-new tier (nonprod or prod) from scratch — one linear pass:

1. cpc — control-plane-core (regional hub VNet, VPN gateway, control-plane KV)
2. dns-zones — control-plane-dns-zones (global private DNS zones in rg-eshars-global-{tier})
3. dns — control-plane-dns (the dnsmasq forwarder VM)
4. wn — workload-network (the spoke VNet for the tier)
5. wc — workload-core (tier RGs, staging storage, App Service Plan, App Gateway, ReportViewer tier KV — created empty)
6. ops — the ops VM (must precede wc-secrets, rs, and js — wc-secrets runs on its runner; rs/js grant its identity Run Command)
7. wc-secrets — workload-core-secrets (in-spoke data-plane writes to the RV KV: EncryptionKey/EncryptionkeyByte + reportviewer-ssl-cert; must run from inside the spoke because the vault is private-endpoint-only)
8. rs and js — the report-server and job-scheduler VMs (either order)
9. env {env} — the first environment
10. Post-apply wiring — the four scripts (see below), ending with the readiness gate

No re-apply pass. The old “apply workload-core again after report-server” two-pass wrinkle is gone: the App Gateway’s SSRS backend is a static IP (ssrs_backend_ip in tfvars), the ReportViewer-KV grants for the SSRS VM live on the report-server root, and SSRSDomain flows rs → env. The estate is a single linear chain.

Adding another environment to a tier that already exists:

1. env {env} + the post-apply wiring. That’s it.

Key idea: steps 1–8 are done once per tier, not once per environment. All nonprod environments (dev, qa, regression) share one nonprod hub, forwarder, spoke network, core, and the tier VMs. All prod environments (uat, training, prod) share the prod ones.

Why this exact order

Step	Root	What it builds	Why it must come first
1	control-plane-core (cpc)	The regional hub VNet, subnets, NSGs, VPN gateway, control-plane KV	Everything peers to this hub. One hub per region (-{tier}-{region}).
2	control-plane-dns-zones (dns-zones)	The global private DNS zones (privatelink.blob…, privatelink.vaultcore…, privatelink.redis…, privatelink.azconfig…) + hub VNet links, in the region-less rg-eshars-global-{tier}	Reads cpc’s vnet_id to link. Workloads register private-endpoint records into these zones. (The SQL zone privatelink.database… is owned by workload-network.)
3	control-plane-dns (dns)	The dnsmasq forwarder VM, static IP cidrhost(vms_subnet_cidr, 4)	Reads cpc’s vms subnet via remote state. Split out so the churny VM has its own lifecycle.
4	workload-network (wn)	Spoke VNet + subnets (vms, private-endpoint, appgw), NSGs, hub peering, SQL private DNS zone	Reads hub coordinates from cpc; everything after looks this VNet up by name.
5	workload-core (wc)	Tier RGs, staging storage account, App Service Plan, App Gateway (static backends incl. ssrs_backend_ip), ReportViewer tier KV (kv-eshars-rv-{tier}-{region}) + the tier-wide encryption key, automation account, BACPAC handoff	workload-env and the VM roots read RGs/plan/storage from here. Reads no VM state — that’s what keeps the chain linear.
6	ops (ops)	The ops VM — in-VNet deploy runner (self-hosted GitHub runner) + its managed identity	rs and js grant this VM’s MI Virtual Machine Contributor (Run Command) for CD deploys. The grant is null-gated on ops state: apply rs/js before ops and the grant is silently skipped — CD then fails with AuthorizationFailed until you re-apply them. Apply ops first and it never bites. wc-secrets also runs on this VM’s runner.
7	workload-core-secrets (wc-secrets)	The three RV-KV secret values: EncryptionKey/EncryptionkeyByte (read from wc’s outputs — the key stays anchored in wc state) and reportviewer-ssl-cert (copied from the scaffold wildcard PFX)	The RV KV is private-endpoint-only, so these are data-plane writes that succeed only from in-spoke — wc (which runs hosted, off-spoke) creates the vault empty; this root fills it from the ops runner. Must come after ops.
8	report-server (rs), job-scheduler (js)	The SSRS VM (one per tier; hosts SSRS + the ReportViewer app) and the jobs VM	rs also grants the SSRS VM’s MI access to the ReportViewer tier KV (looks the wc-created vault up by name). Either order; both after ops.
9	workload-env (env)	Everything per-environment: per-env web + function app, SQL server + DBs, storage, Key Vault + secrets, private endpoints, DNS alias	The leaf. Reads wc (plan/RGs), rs (ssrs_url, computer_name → SSRSDomain, SSRS creds), ops (deploy grants).

Graceful bootstrap (what happens if you apply env before rs)

workload-env reads report-server outputs through a remote-state defaults block — if rs hasn’t been applied, they resolve to null instead of failing the plan. The SSRS-dependent pieces then simply drop out: the KV grants are null-gated, and null app-setting values (SSRSDomain, MvcReportViewerReportServerUrl) are dropped by map(string) coercion. Nothing crashes — but reporting is unwired until you re-apply env after rs exists. Follow the linear order and this never happens.

Dependency map

flowchart TB
    CPC["control-plane-core (cpc)<br/>hub VNet + VPN gateway"]
    DZ["control-plane-dns-zones<br/>global privatelink zones"]
    DNS["control-plane-dns<br/>forwarder VM (.4)"]
    WN["workload-network (wn)<br/>spoke VNet + subnets"]
    WC["workload-core (wc)<br/>RGs, ASP, App GW, RV tier KV (empty)"]
    OPS["ops VM<br/>deploy runner + MI"]
    WCS["workload-core-secrets (wc-secrets)<br/>in-spoke RV-KV writes"]
    RS["report-server (rs)<br/>SSRS VM + RV app"]
    JS["job-scheduler (js)<br/>jobs VM"]
    ENV["workload-env {env}<br/>apps, SQL, env KV, PEs"]

    CPC -- vnet id --> DZ
    CPC -- vms subnet --> DNS
    CPC -- hub coords --> WN
    WN --> WC
    WC --> OPS
    OPS -- "runs on its runner" --> WCS
    WC -- "tier key + RV KV id" --> WCS
    OPS -- "ops MI (Run Command grant)" --> RS
    OPS -- "ops MI (Run Command grant)" --> JS
    WC -- "RV KV looked up by name" --> RS
    RS -- "ssrs_url, computer_name, creds" --> ENV
    WC -- "plan, RGs, tier key" --> ENV
    OPS -- "deploy grants" --> ENV

Auxiliary VM roots (tmhp, sftp-vm, rns, successed, db-api) follow the same pattern as rs/js: after ops, reading wc/wn by name. They’re optional per tier and not part of the core chain.

Prerequisites (once, before anything)

1. Terraform backends — tier state storage accounts (sattf{tier}{region} in rg-eshars-scaffold-{tier}-{region}). See BACKEND_SETUP.md.
2. Scaffold KV seeded (kv-eshars-scaf-{tier} — created by the backend setup, outside the TF estate):
   • the wildcard TLS cert myeshars-wildcard (App Gateway + ReportViewer SSL),
   • the 8 shared-external vendor secrets (canonical copies — copy-kv-secrets.sh refuses to run against an unseeded scaffold). See the architecture page.
3. No TF_VAR_* secret exports. There are no sensitive Terraform inputs left anywhere in the estate (eshars-ops #79/#81). The two operator-supplied values live as scaffold-KV secrets, seeded once per tier: SqlServerAdminLogin (the admin login base — do not pre-suffix it; workload-env appends the env itself, and the variable defaults to eshars-sql-admin if unset) and TeamsWebhookUrl (may be blank). The Infra: 6 workflow reads both from kv-eshars-scaf-{tier} at run time. SQL admin passwords and VM admin passwords are never inputs — Terraform generates them and writes them to the appropriate Key Vault. See the Workload-env Stand-up runbook prerequisites for the seeding commands.
4. Run from inside the spoke (ops runner) or on the VPN with working split-DNS. The env KV is private-endpoint-only: applying workload-env from a machine whose DNS resolves the vault to its public endpoint fails secret writes with 403 ForbiddenByConnection.

Commands

Prefer just over raw tofu — the wrappers set the correct backend config. cpc/dns-zones/dns/wn/wc/wc-secrets/ops/rs/js take a tier (nonprod/prod); only env takes an environment token (dev, qa, regression, uat, training, prod).

# Brand-new nonprod tier, from scratch — one linear pass:
just init-cpc nonprod        && just plan-cpc nonprod        && just apply-cpc nonprod
just init-dns-zones nonprod  && just plan-dns-zones nonprod  && just apply-dns-zones nonprod
just init-dns nonprod        && just plan-dns nonprod        && just apply-dns nonprod
just init-wn  nonprod        && just plan-wn  nonprod        && just apply-wn  nonprod
just init-wc  nonprod        && just plan-wc  nonprod        && just apply-wc  nonprod
just init-ops nonprod        && just plan-ops nonprod        && just apply-ops nonprod
just plan-wc-secrets nonprod && just apply-wc-secrets nonprod   # in-spoke only (ops runner / VPN); recipe inits itself
just init-rs  nonprod        && just plan-rs  nonprod        && just apply-rs  nonprod
just init-js  nonprod        && just plan-js  nonprod        && just apply-js  nonprod
just init-env dev            && just plan-env dev            && just apply-env dev

# Adding another environment to an existing tier (e.g. qa):
just init-env qa && just plan-env qa && just apply-env qa

Post-apply wiring (after every apply-env)

Terraform creates the credential material; these scripts apply it where Terraform can’t reach (SQL logins, vendor values, running apps). Run them in order — full detail in the Workload-env Stand-up runbook:

# 1. Fan the 8 shared-external vendor secrets from the scaffold KV into the env KV
./scripts/copy-kv-secrets.sh <env>
az webapp restart --name app-eshars-<env>-<region> --resource-group rg-eshars-web-<tier>-<region>  # app caches KV reads

# 2. Create the low-privilege SSRS report-execution SQL login (matches the KV secret)
./scripts/setup-db-ssrs-reader.sh <env> rg-eshars-data-<tier>-<region>

# 3. Apply the database-level Entra group grants (reader / writer / dba)
./scripts/setup-db-entra-groups.sh <env> rg-eshars-data-<tier>-<region>

# 4. THE GATE — the environment is not ready until this exits 0
./scripts/verify-env-secrets.sh <env>

The gate fails (exit 1) while any of its 9 gated secrets (the 8 vendor secrets + Jwt--Secret) still holds PLACEHOLDER-*, is missing, or is unreadable — it fails closed. A leftover placeholder is a silent runtime failure (vendor 401), so do not declare the environment up while the gate fails. (ImportApiUserPassword, AzureFunctionsHostKey, and masterPassword are deliberately not gated — see the placeholder table for each reason; in particular never seed masterPassword.)

Teardown

Reverse of the build order:

# Workload chain first (leaf inward):
#   env → js / rs → wc-secrets → ops → wc → wn
# Then the control plane — dns BEFORE cpc, so the forwarder VM tears down
# without disturbing the hub VNet, gateway, or zones:
just destroy-dns nonprod
just destroy-cpc nonprod

Key Vault soft-delete holds names for 7 days. Rebuilding immediately hits 409 Conflict on env/RV vault names — purge first (az keyvault purge --name <kv> --location centralus). The scaffold KVs are purge-protected by design and are never torn down.

Notes

• Always plan before apply. Read the plan — reject anything that destroys or changes resources you didn’t expect.
• All changes go through PR. No manual portal changes.
• The application estate (web app code, job scheduler, reports, ReportViewer) is deployed separately by the HISD repo’s tier-aware CD — see the architecture page.