Skip to content
eSHARS Operations

INFRA-0010: Cross-Root State, App-Setting Ownership, and the RBAC Module

Status

Accepted — 2026-06-22

Date

2026-06-22

Context

The rebuilt estate is split into many OpenTofu roots (control-plane-core, control-plane-dns, control-plane-dns-zones, workload-network, workload-core, workload-env, and the vms/* roots), each with its own state. Roots communicate via terraform_remote_state. Three recurring failure modes during the rebuild all trace back to which root owns a given value and how stored state propagates between roots. This decision records the rules that resolve them.

Decision

1. Each app setting has exactly one owning root

The tier-level web app and function app are created by workload-core (via modules/web-app / modules/function-app); the per-env slots and their per-env settings are created by workload-env. When both roots write the same app_settings key, they fight on every apply — each apply reverts the other.

Rule: a per-env value is owned by workload-env; workload-core must not write it at all. Concretely, for keys whose value is per-env (e.g. SecureSettingsKeyVault = the per-env Key Vault name):

  • modules/web-app emits the key only when its input is non-null (workload-core passes key_vault_name = null → the key is omitted from core’s output entirely — it does not write an empty string).
  • The key is in the root app’s lifecycle.ignore_changes, so even a future workload-core apply can never blank workload-env’s value.
  • workload-env is the sole writer, on the slot, with the real per-env KV name.

This is the general pattern for any shared key. Tier-invariant keys (App Insights extension version, time zone) may live in core; anything that varies per environment belongs to env.

2. terraform_remote_state reads stored state, not code

A consumer root reads the last-applied outputs of the producer root, not the producer’s current source. Therefore:

Adding or changing an output in a producer root has no effect on consumers until the producer is re-applied (which rewrites its state blob). A consumer planned against the old state sees the old/absent value.

Two concrete bites from the rebuild:

  • The SSRS App Gateway backend (workload-core) is gated on report-server’s private_ip_address. It registered only after report-server’s state had the IP and workload-core was re-applied to re-read it.
  • The jobs-MI Key Vault grant (workload-env) is gated on job-scheduler’s principal_id output. The grant silently skipped because job-scheduler’s state had outputs: [] (written before that output existed in code). Fix: re-apply job-scheduler (0 resource changes — it just writes outputs to state), then re-apply workload-env.

Operational rule: after adding an output to a root, re-apply that root before expecting any downstream consumer to see it.

3. RBAC is a shared module fed a per-root grant list

modules/rbac takes grants = list({ key, scope, role, principal_id }), for_each’s azurerm_role_assignment, and filters out null-principal entries. This replaces the old count = principal == null ? 0 : 1 gates with a single, uniform pattern.

  • Instantiated once per root — state is per-root, so a single estate-wide grant map is not possible.
  • A null principal_id (e.g. a remote-state principal not yet populated — see rule 2) means that grant is simply skipped this apply. This is the graceful-bootstrap behaviour: applies don’t fail when a dependency isn’t ready, but the grant will not exist until the principal is non-null and the root is re-applied. It is silent — no error prompts the re-apply.
  • Migration onto the module used moved {} blocks so the ~38 live grants moved in state with 0 destroy (verified — recreating a grant would briefly revoke access on a running estate).

Some grants stay inline (they don’t fit a flat list): conditional grants gated on a feature flag (deploy_app_gateway), self-referential grants, slot fan-out grants (per-slot KV/storage, driven by the app identity not the deploy identity), and module-internal grants.

Known gaps and follow-ups (from the 2026-06-22 estate-wide scan)

A four-way scan (remote-state wiring, legacy references, count-gate skips, app-setting/hostname alignment) found the nonprod rebuild clean on the active path. Items still open:

FindingStatusAction
Two RBAC gaps: ops MI Website Contributor on the web app; jobs MI KV Secrets User per-envClosedBoth grants added; ops grant verified live
scalr/main.tf hardcodes the legacy saesharstfstate for every workspace backendClosed (2026-07)Repointed: the scalr root now maps each tier to its own account (sattfnonprodcus / sattfprodcus), and its own state lives in sattfprodcus
vms/tmhp-sftp/environments/prod.tfvars references legacy rg-eshars-prod / db-eshars-prod / kv-eshars-tmhpDeferred (prod round)Correct for now — legacy prod SQL/KV are still live; tmhp-sftp prod not yet applied
vms/tmhp-sftp/main.tf unguarded private_dns_zone_ids[...] map subscriptOpen (low)Wrap in try(); bundle with prod-round tmhp work
Prod-tier App Gateway / report ingress not deployedDeferred (prod round)deploy_app_gateway = true + appgw_backends/ssrs_host_name for prod
Legacy VNet peerings, custom-DB legacy DNS linkBy designMigration bridges; remove post-cutover
github_sp_* OIDC grantsBy design — now permanentSuperseded by the foundation/workload identity split (eshars-ops #79): the app CD auths via the ops VM MI, but the infrastructure workflows deliberately auth as two OIDC SPs (github-eshars-platform, github-eshars-deploy) — the grants stay. See Stand-up Workflows

Consequences

  • Plans that show “0 changes” when you expected a new grant/backend are usually a stale-state problem (rule 2), not a code bug. Re-apply the producer first.
  • A grant silently missing at runtime (a 403 from an app/MI) is usually a null-principal skip (rule 3). Check the producer’s state has the principal output, then re-apply.
  • App-setting churn on every apply means a shared-ownership key (rule 1) escaped the omit-or-ignore_changes pattern.
  • INFRA-0003 — slot Key Vault + App Gateway pattern
  • INFRA-0004 — dual app-setting keys for MvcReportViewer
  • Report Server — consumes the cross-root + AG-backend pattern