Configuration & Secrets Architecture

How every piece of configuration — connection strings, vendor API keys, SSRS credentials, feature flags — gets from where it lives to the code that reads it. This is the WHY page; the step-by-step for standing infrastructure up is the Deployment Order runbook, and the per-environment procedure is the Workload-env Stand-up runbook.

Everything on this page is verified against eshars-ops develop and the HISD repo branch ci/tier-aware-deploys-v2 (July 2026).

The model in one paragraph

Terraform creates a per-environment Key Vault with everything it can derive (connection strings, generated passwords) and PLACEHOLDER-* stubs for everything it can’t (vendor API keys, crypto master password). A per-tier scaffold Key Vault — living outside the Terraform estate so it survives teardown — is the canonical source for the shared vendor secrets, fanned into each env KV by copy-kv-secrets.sh. A per-tier ReportViewer Key Vault holds the tier-wide report encryption key and SSRS credentials for the one-per-tier ReportViewer app. At runtime, each .NET app pulls its secrets from its vault via a greedy Key Vault config builder (selected by the SecureSettingsKeyVault setting), layered on top of non-secret App Service application settings. The tier-aware CD pipeline stamps the vault names, environment labels, and SSRS data sources at deploy time.

flowchart TB
    subgraph sources["Sources of truth"]
        SCAF["kv-eshars-scaf-{tier}<br/>(scaffold KV — outside TF,<br/>survives teardown)"]
        TF["Terraform<br/>(workload-core + workload-env)"]
        GHV["GitHub Environment vars<br/>(APP_SERVICE_NAME, DB_SERVER…)"]
    end

    subgraph vaults["Runtime vaults"]
        ENVKV["kv-eshars-{env}-{region}<br/>(per-env KV)"]
        RVKV["kv-eshars-rv-{tier}-{region}<br/>(tier ReportViewer KV)"]
    end

    subgraph apps["Consumers"]
        WEB["Web app + Function app<br/>app-eshars-{env}-{region}"]
        JOBS["Job scheduler service<br/>(jobs VM)"]
        RV["ReportViewer IIS app<br/>(SSRS VM, one per tier)"]
        SSRS["SSRS data sources<br/>(connect to SQL as SSRSReader)"]
    end

    TF -- "derived values +<br/>PLACEHOLDER stubs" --> ENVKV
    TF -- "tier encryption key,<br/>SSRS creds, SSL cert" --> RVKV
    TF -- "same tier key fanned<br/>into every env KV" --> ENVKV
    SCAF -- "copy-kv-secrets.sh &lt;env&gt;<br/>(8 vendor secrets)" --> ENVKV
    SCAF -- "wildcard cert copied<br/>by workload-core" --> RVKV

    ENVKV -- "greedy KV config builder<br/>(SecureSettingsKeyVault)" --> WEB
    ENVKV -- "greedy KV config builder" --> JOBS
    RVKV -- "greedy KV config builder<br/>+ SSL cert import" --> RV
    ENVKV -- "deploy-reports.ps1 stamps<br/>SSRSReader cred" --> SSRS
    GHV -- "App Service app settings /<br/>deploy script parameters" --> WEB

Two channels into an app

Every .NET Framework app in the estate receives configuration through exactly two channels, layered by config builders declared in its Web.config / App.config:

Channel	Carries	Mechanism
App Service application settings (environment variables)	Non-secret config: Environment, WebApiBaseUrl, SSRSReportServerUrl, SSRSDomain, SSRSReportFolder, container names, email/SFTP flags, fileStoragePath…	Terraform web_app_settings_defaults (+ per-env web_app_settings_override) in workload-env/locals.tf → App Service settings → injected as env-vars → the Environment config builder overwrites matching <add key> entries
Key Vault secrets	Everything sensitive: connection strings, vendor API keys, SSRS credentials, encryption keys	The AzureKeyVault config builder in greedy mode reads every secret in the vault named by SecureSettingsKeyVault and overwrites matching config entries

Two rules make this predictable:

1. The greedy builder is placeholder-driven. A KV secret only lands in the app if the config file already declares a same-named empty entry (<add key="EchoSignAppKey" value="" />). Secrets with no matching placeholder are silently ignored — the vault can hold more than any one app consumes.
2. Builder order is Environment,AzureKeyVault — env-vars apply first, Key Vault last. On a name collision, the KV secret wins. This is why the ReportViewer deploy script must not seed an Environment secret into the RV vault: the greedy builder would override the value the script patched.

SecureSettingsKeyVault is itself an app setting: it names the vault the greedy builder reads (vaultName="%SecureSettingsKeyVault%"). Terraform sets it on the App Service; the CD pipeline additionally substitutes the token into the published config. Point an app at a different vault and it reads a different secret set — that is the entire per-env / per-tier isolation mechanism.

Apps authenticate to their vault with their managed identity (App Service MI or VM MI), granted Key Vault Secrets User in Terraform. No stored credentials anywhere in the chain.

The three Key Vaults

	Per-env KV	Tier ReportViewer KV	Scaffold KV
Name	kv-eshars-{env}-{region}	kv-eshars-rv-{tier}-{region}	kv-eshars-scaf-{tier}
Created by	workload-env (modules/key-vault-secrets)	workload-core (reportviewer-kv.tf)	Out-of-band (setup-terraform-backends.sh), in rg-eshars-scaffold-{tier}-{region}
Inside the TF estate?	Yes — destroyed on env teardown	Yes — destroyed on tier teardown	No — survives teardown/rebuild
Purge protection	Off (name reclaimable on rebuild)	Off	On — the only purge-locked vaults
Holds	TF-derived secrets + 12 placeholder stubs (below)	ReportServerUsername/Password, tier-wide EncryptionKey/EncryptionkeyByte, reportviewer-ssl-cert	Wildcard TLS cert myeshars-wildcard, storage CMK, GitHub-runner app creds, the 8 shared-external vendor secrets (canonical copies)
Read by	Web app, function app, job scheduler (greedy builders); deploy scripts (SqlAdminPassword, SSRSReader*)	ReportViewer IIS app on the SSRS VM	copy-kv-secrets.sh (fan-out), App Gateway (cert), workload-core (copies cert → RV vault)
Network	Private endpoint only	Private endpoint only	Public (bootstrap vault — must be reachable before any private networking exists)

Why each vault exists:

• Per-env — each environment’s apps must only ever see that environment’s connection strings and credentials. SecureSettingsKeyVault=kv-eshars-dev-cus on the dev app is the isolation boundary.
• Tier RV — the ReportViewer app is deployed once per tier, not per environment, so it can’t read a per-env vault. It gets its own tier-scoped vault. Its MI grants (rv-mi-secrets-user, rv-mi-certificate-user) deliberately live on the report-server root, not workload-core — the report-server root knows the SSRS VM’s identity directly and looks the vault up by name, which keeps the estate a single linear apply (no second workload-core pass to “pick up” the VM identity).
• Scaffold — some secrets must outlive any Terraform teardown: the wildcard TLS cert, and the vendor API keys that no rebuild can regenerate. The scaffold vault is the durable, canonical home; everything else is derivable or re-seedable from it. Seeded once per tier (nonprod was seeded from the legacy kv-eshars-qa on 2026-07-07).

The tier-wide encryption key

EncryptionKey / EncryptionkeyByte encrypt PII columns and the ReportViewer ?i= launch blob (next section). Because the ReportViewer is one app per tier and must decrypt blobs produced by every environment’s web app, the key must be tier-wide: workload-core generates it once (random_uuid + random_bytes, anchored in wc state), the workload-core-secrets root writes it into the RV vault (an in-spoke data-plane write — the vault is private-endpoint-only, so off-spoke wc creates it empty), and workload-env writes the same value into every per-env KV. A per-env key would mean a report launched from qa couldn’t be decrypted by the tier RV app.

Crypto keys on a migrated-data environment

EncryptionKey values are baked into existing encrypted DB data. For any environment restored from real data, the key must match the value that encrypted that data — pin via tier_encryption_key_override before the apply. A fresh key silently makes existing encrypted data unreadable: no error at deploy time, just undecryptable PII at read time. Fresh/synthetic environments can take generated values.

(masterPassword is not a crypto key and is not part of this — see the placeholder table below.)

Secret lifecycle: the placeholder pattern

modules/key-vault-secrets creates every env-KV secret at apply time. Secrets Terraform can derive get real values; the rest are created as PLACEHOLDER-update-after-provisioning with Placeholder="true" tags and lifecycle { ignore_changes = [value] } — so once a real value is written, future applies leave it alone.

TF-derived (never placeholders): the five connection strings (ESHARS, ESHARSLOGS, ESHARSAUDIT, ESHARSREPORTS, ESHARSREADONLY), RedisConnectionString, blob connection strings, applicationInsightKey, SqlAdminPassword, MvcReportViewerUsername/Password + ReportServerUsername/Password (from report-server remote state — same SSRS account, two name pairs; see INFRA-0004), SSRSReaderUsername/Password (Terraform-generated random_password), EncryptionKey/EncryptionkeyByte (the tier key).

The remaining placeholders / special-cased secrets, and what fills each:

Secrets	Filled by
EchoSignAppKey, SendGridApiKey, TMHPServerKey, zendeskSharedKey, HandShakingKey, DevOpsTeamsWebhookURL, SystemHealthNotificationWebhookURLEligibility, SystemHealthNotificationWebhookURLImports	copy-kv-secrets.sh <env> — fans the canonical values from the scaffold KV; refuses to run if the scaffold copy is missing or itself a placeholder
Jwt--Secret	TF-generated (random_password.jwt_secret in modules/key-vault-secrets). It is the eshars-mobile-api’s HMAC-SHA256 signing key (KV Jwt--Secret → config Jwt:Secret) — not consumed by HISD, but in use by the mobile API, so it is not removable. Never hand-seed or delete it
ImportApiUserPassword	Manual, when provisioning the import user — must match a real eSHARS Identity user’s password (imports authenticate via OAuth password-grant as that user), so TF cannot generate it. Import-feature setup, not a stand-up step; until set, imports 401 (imports-only, non-fatal)
AzureFunctionsHostKey	Deploy-time read-back — the function app’s own host key does not exist until HISD’s first function deploy materializes it; the app CD writes it into the env KV post-deploy. Until set, WebUI grid-settings/audit 401 (non-fatal)

masterPassword is deliberately NOT provisioned

masterPassword is not a crypto key — it is a universal auth backdoor. HISD’s auth paths treat any non-empty value as a master override login: a failed password check is retried against masterPassword, and a match logs in as any user (including PHI access). The app guards on IsNotNullAndEmpty(), so the safe state is the secret being absent — modules/key-vault-secrets no longer creates it, and verify-env-secrets.sh deliberately does not gate it (gating would push operators to seed a value, i.e. create the backdoor). Never seed, copy, or “pin” a value. The feature itself is flagged for security review/removal (see eshars-security findings).

The gate: scripts/verify-env-secrets.sh <env> checks a fixed list of 9 secrets by name — the 8 vendor secrets plus Jwt--Secret (TF-generated, kept as a backstop) — against the live vault and exits 1 if any still holds a PLACEHOLDER-* value (or is missing/unreadable — it fails closed). An environment is not stood up until the gate passes. This exists because every one of these is a silent runtime failure, not a deploy error: the app happily starts and then 401s against Adobe Sign, or serves undecryptable data. (ImportApiUserPassword, AzureFunctionsHostKey, and masterPassword are deliberately not gated — see above for each reason.)

The ReportViewer chain

Reports are rendered by a standalone IIS WebForms app (Eshars.ReportViewer) on the SSRS VM — one instance per tier — hosting the Microsoft.Reporting.WebForms ReportViewer control. End-to-end:

sequenceDiagram
    participant U as Browser
    participant W as Web app (per env)
    participant RV as ReportViewer app (per tier)
    participant S as SSRS (localhost on same VM)
    participant DB as Azure SQL (per env)

    U->>W: open report
    W->>W: build {ENV, RNM, REID, TS} JSON<br/>encrypt AES-128-CBC with tier EncryptionKey
    W-->>U: redirect to {SSRSReportServerUrl}/ReportViewer.aspx?i={blob}
    U->>RV: GET ?i=…
    RV->>RV: decrypt with the SAME tier key,<br/>validate timestamp/env/path
    RV->>S: render /{env}/Reports/{name}<br/>(ReportServerUsername/Password)
    S->>DB: dataset queries as SSRSReader<br/>(low-priv, db_datareader + EXECUTE rpt*)
    S-->>RV: rendered report
    RV-->>U: HTML/PDF

Key facts that are easy to get wrong:

• SSRSReportServerUrl is the ReportViewer web app’s base URL (fronted by the App Gateway at npreports.myeshars.com / reports.myeshars.com) — it is not SSRS. The web app builds {SSRSReportServerUrl}/ReportViewer.aspx?i={encrypted}.
• The ?i= blob is {ENV, RNM (report path), REID (execution id), TS (expiry ticks)} encrypted with AES-128-CBC using EncryptionKey (key) and EncryptionkeyByte (IV). Encrypt side = each env’s web app; decrypt side = the one tier RV app — hence the tier-wide key.
• Two SSRS credential families exist — don’t mix them up:
  • The ReportViewer app authenticates to SSRS with ReportServerUsername/ReportServerPassword from the RV tier vault (empty domain — it talks to SSRS on localhost).
  • The web app and job scheduler (SA-form direct-render path, MvcReportViewer) authenticate with MvcReportViewerUsername/MvcReportViewerPassword + SSRSDomain from the per-env vault / app settings, calling http://10.x.1.6/ReportServer directly across the network (the App Gateway can’t proxy SSRS auth).
• SSRSDomain must be the VM’s NetBIOS computer name (ssrs-np-cus), NOT the Azure resource name (vm-eshars-ssrs-nonprod-cus, which exceeds the 15-char NetBIOS limit). Terraform sources it from the report-server root’s computer_name output. The wrong value produces a hard 401 on every render.
• ReportServerRoutingMode toggles the RV app between Legacy (the committed default: one RV fanning out to per-environment SSRS URLs — QAReportServerUrl etc.) and Tier (the Azure model: single ReportServerUrl=http://localhost/ReportServer, report path prefixed /{env}/…). deploy-reportviewer.ps1 patches it to Tier on every automated deploy, so Azure estates always run tier routing.
• SSRS itself connects to SQL as SSRSReader — a dedicated low-privilege SQL login (db_datareader + EXECUTE on the rpt* procs only), stamped onto the shared data sources (HisdDbDataSource, HisdDbReportDataSource, HistReportDbDataSource) by deploy-reports.ps1. The matching SQL login is created by setup-db-ssrs-reader.sh from the same KV secrets, so data source and login can never drift apart. Reports never run as the SQL admin.

What the tier-aware CD stamps at deploy time

The HISD repo’s CI/CD (ci-build.yml + cd-deploy.yml, branch ci/tier-aware-deploys-v2) deploys the whole application estate. Structural facts:

• Runs on a self-hosted runner on the tier’s ops VM (runs-on: [self-hosted, eshars, {tier}]), authenticating with the VM’s system-assigned managed identity (az login --identity) — no OIDC service principal for the app CD. This is a hard requirement: every data-plane endpoint (Key Vaults, storage, SQL, App Service SCM) is private-endpoint-only, so deploys must originate inside the VNet. (The eshars-ops infrastructure workflows are different — they authenticate as two OIDC service principals, github-eshars-platform and github-eshars-deploy; see Stand-up Workflows.)
• Slot → tier resolution: dev/qa/regression → nonprod; uat/training/prod → prod. (staging is deliberately absent: it is an infra-tier environment in workload-env — nonprod, per infra_tier — but cd-deploy.yml has no staging deploy slot. Don’t confuse it with prod’s staging deployment slot in the next bullet, which is a swap mechanism, not an environment.) All resource names are derived from the CAF convention; per-env values (App Service name, DB server, SQL user) come from GitHub Environment variables.
• Web app: deployed to the per-env App Service; the pipeline sets SecureSettingsKeyVault as an app setting and substitutes the token in the published Web.config. Prod deploys to the staging slot and swaps for zero-downtime.
• VM-hosted components (job scheduler, DbUpgrader, TMHP, ReportViewer, reports): artifacts are staged to the tier staging storage account (saeshars{tier}cus), container deploy-artifacts, blob prefix {component}/{ci-run-id}/, then executed on the target VM via managed Run Command (real exit-code polling; scripts must be pure ASCII — RunCommand corrupts non-ASCII).
• Ordering that matters: wake-db first (resumes serverless SQL so nothing times out), then deploy-dbupgrader before deploy-jobscheduler (schema migrations land before the scheduler starts against the DB).
• Per-component config stamping:
  • deploy-reportviewer.ps1 — patches the RV Web.config: vault name → the tier RV KV, ReportServerUrl → http://localhost/ReportServer, ReportServerRoutingMode → Tier, Environment → the tier label; imports the SSL cert from the RV vault (reportviewer-ssl-cert, base64 PFX) and binds HTTPS.
  • deploy-jobscheduler.ps1 — substitutes #{SecureSettingsKeyVault}# and #{JobSchedulerEnvironment}# tokens; installs the per-slot TopShelf service.
  • deploy-reports.ps1 — publishes report definitions and stamps the SSRS data sources with the SSRSReader credential from KV.
  • deploy-dbupgrader.ps1 — pulls SqlAdminPassword from KV; passes it via the DBUPGRADER_PASSWORD env-var so it never appears in a process command line.

Failure signatures

Symptoms → root cause, from live incidents (July 2026 dev rebuild):

Symptom	Root cause	Fix
Vendor API returns 401 INVALID_ACCESS_TOKEN (Adobe Sign, SendGrid…)	Env-KV vendor secret still PLACEHOLDER-* — nothing fanned the scaffold values in	./scripts/copy-kv-secrets.sh <env>, restart the app
SSRS render returns 401 on GET /ReportServer (App Insights dependency)	SSRSDomain holds the Azure resource name instead of the NetBIOS computer_name	Sourced from the rs root’s computer_name output — re-apply rs then env
Login failed for user 'SSRSReader' in the SSRS log → report 500s	The SQL login was never created on the rebuilt server (KV secrets alone aren’t enough)	./scripts/setup-db-ssrs-reader.sh <env> rg-eshars-data-{tier}-{region}
Report data silently unreadable / decryption errors on migrated data	EncryptionKey regenerated instead of pinned	Pin to the historical value (see danger box) — prevention only; damage needs a re-restore
RV renders but shows the wrong environment’s report/data	ReportServerRoutingMode not patched to Tier, or path prefix missing	Re-run the ReportViewer deploy
App starts but every KV-sourced setting is empty	SecureSettingsKeyVault unset/wrong, or the app MI lacks Key Vault Secrets User	Check app settings + the RBAC grant in workload-env

The catch-all check: ./scripts/verify-env-secrets.sh <env> — if it exits non-zero, the environment is not fully wired, whatever the symptom.

Open items

• Remove the masterPassword override-login feature from HISD entirely (tracked in eshars-security) — the secret is already not provisioned; the code path is what remains.
• TF-generate ImportApiUserPassword (+ SQL user script) — the last truly manual placeholder fill. (AzureFunctionsHostKey is already written back by the app CD post-deploy; Jwt--Secret is already TF-generated.)
• TMHPServerKey in the scaffold KV is 2 characters (harvested as-is from legacy qa) — likely a dummy; confirm with the TMHP integration owner before relying on it.
• Fold copy-kv-secrets.sh + setup-db-ssrs-reader.sh + verify-env-secrets.sh into an apply-env CI pipeline (tracked).

Related

• Deployment Order — the full-estate runbook
• Workload-env Stand-up runbook
• INFRA-0004: Dual App Setting Keys (MvcReportViewer)
• INFRA-0010: Cross-Root State, App-Setting Ownership & RBAC
• Adobe Sign Key Generation