Workload-env Stand-up Runbook

End-to-end procedure to bring up a new eSHARS environment (dev, qa, regression, uat, training, prod) on the tier-based ALZ layout. This is the environment-level chapter of Deployment Order — Standing Up the Estate; the tier-level roots (cpc → dns-zones → dns → wn → wc → ops → wc-secrets → rs/js) must already be applied. The same procedure is automated end-to-end as the Infra: 6 - Environment Workload GitHub Actions workflow — see Stand-up Workflows; this page is the manual path and the reference for what the workflow does.

(staging exists in workload-env’s tier map for historical reasons but has no tfvars and is not a deployable environment — don’t confuse it with prod’s staging deployment slot.) For why configuration and secrets flow the way they do, see Configuration & Secrets Architecture.

When to use this runbook

• Standing up a brand-new environment for the first time
• Rebuilding an environment after a destroy
• Re-verifying an environment that “deployed fine” but misbehaves at runtime (start at step 4c)

Prerequisites

1. Azure CLI authenticated to the eSHARS subscription (9639039c-86c6-4859-b014-d28688863cd5):

   az login
   az account set --subscription 9639039c-86c6-4859-b014-d28688863cd5

2. OpenTofu installed (tofu, not terraform)
3. Working directory at the root of eshars-ops
4. Tier-level roots already applied (once per tier): cpc, dns-zones, dns, wn, wc, ops, rs, js — see the deployment order. Applying an env before rs doesn’t fail, but the SSRS settings null-coerce out and you must re-apply env later — follow the order.
5. Scaffold KV seeded (kv-eshars-scaf-{tier}) with the 8 shared-external vendor secrets — step 4 refuses to run otherwise.
6. Optional env var (only for a manual/laptop apply; the workflow sets its own):

   export TF_VAR_teams_webhook_url='<workflow-url-or-blank>'

   Do NOT export a SQL admin login or password — neither is an operator input:
   • Login: sql_server_admin_login defaults to the canonical base eshars-sql-admin (variables.tf), and local.sql_admin_login appends the env → eshars-sql-admin-<env>. Exporting TF_VAR_sql_server_admin_login (especially an already-suffixed value like eshars-sql-admin-dev) is unnecessary and, if suffixed, doubles to eshars-sql-admin-dev-dev. Leave it unset.
   • Password: Terraform generates it (random_password.sql_admin) and writes it to the env KV (SqlAdminPassword). There is no TF_VAR_sql_server_admin_password; exporting one does nothing. Retrieve the generated value from the KV if you need it.
7. Run from inside the spoke (the ops runner) or on the VPN with working split-DNS. The env KV is private-endpoint-only: a machine that resolves the vault to its public endpoint fails the secret writes with 403 ForbiddenByConnection. On a brand-new env the vault name may still be cached to its public IP when the first secret writes fire, so the apply can 403 even from inside the spoke with the PE-DNS/RBAC polls green — a re-apply clears it once the record has propagated. See Open items / known gaps.
8. GitHub repository environment created for this env in the HISD repo. The env vars/secrets (SQL_ADMIN_PASSWORD, DB_SERVER, …) are Terraform-managed, but the environment itself is not — a GitHub App token can’t create it, so it’s a one-time user-PAT bootstrap. Skipping this makes the apply fail with POST .../environments/<env>/variables: 404 Not Found:

   # gh authenticated as a USER with admin on HISD (App tokens can't do this)
   ./scripts/setup-github-environments.sh <env>          # idempotent (create-or-noop)

Stand-up sequence

1. Initialize the workload-env root for this environment

just init-env dev

Targets workload-env-<env>.tfstate in the tier state account (sattf{tier}{region} in rg-eshars-scaffold-{tier}-{region}). Each environment has its own state file; nonprod CI/CD cannot reach prod state.

2. Plan and review

just plan-env dev

Expected for a brand-new env: ~40-50 resources to add, 0 to change, 0 to destroy. Walk the plan output and confirm:

• All resources land in the correct tier RGs — convention is rg-eshars-{component}-{tier}-{region} (component before tier), e.g. rg-eshars-data-nonprod-cus, rg-eshars-web-nonprod-cus
• SQL Server is per-env (sql-eshars-<env>-<region>)
• Web app and function app are per-env root apps — app-eshars-<env>-<region>, func-eshars-<env>-<region> — on the tier-shared App Service Plan (asp-eshars-{tier}-{region}). No per-env slots; only prod gets a staging slot (zero-downtime swap)
• The env Key Vault is kv-eshars-<env>-<region> with a private endpoint
• Private DNS: the SQL alias zone (db.<env>.eshars.internal) lands in the network RG (rg-eshars-network-{tier}-{region})

3. Apply

just apply-env dev

Typical duration: 8-12 minutes (SQL Server creation is the slow step, ~5 minutes).

Two first-apply failure modes on a brand-new env (both prerequisites above):

• POST .../environments/<env>/variables: 404 Not Found — the GitHub repository environment was never created. Run setup-github-environments.sh <env> (prerequisite 8), then re-apply.
• 403 ForbiddenByConnection on the env-KV secret writes despite green PE-DNS/RBAC polls — the new vault name was still cached public. Re-run just apply-env <env>; it succeeds once the record has propagated (prerequisite 7).

4. Populate shared-external secrets from the scaffold KV

workload-env creates the shared-external secrets (vendor API keys, e-sign key, notification webhooks) with PLACEHOLDER-update-after-provisioning values under lifecycle { ignore_changes = [value] }. A rebuilt env KV therefore comes up with placeholders — if nothing copies the real values in, the app authenticates to Adobe Sign / SendGrid / TMHP with the literal placeholder and the vendor returns 401 INVALID_ACCESS_TOKEN (this is exactly how the dev SAForm document-upload broke on the 2026-07 rebuild).

Canonical source = the per-tier SCAFFOLD Key Vault (kv-eshars-scaf-{tier}). The scaffold KV lives outside the Terraform estate (rg-eshars-scaffold-{tier}-{region}) so it survives teardown/rebuild — it is the source-of-truth for these values. copy-kv-secrets.sh derives the source scaffold KV from the env’s tier automatically:

# Dry run first — resolves the 8 shared-external secrets, refuses any placeholder
./scripts/copy-kv-secrets.sh --dry-run dev

# Then for real  (kv-eshars-scaf-nonprod -> kv-eshars-dev-cus)
./scripts/copy-kv-secrets.sh dev

Tier is derived from the env: dev/qa/regression/staging → nonprod; uat/training/prod → prod (in lockstep with infra_tier). After writing the real values, restart the app so it re-reads the vault:

az webapp restart --name app-eshars-<env>-<region> --resource-group rg-eshars-web-<tier>-<region>

The script copies 8 shared-external secrets (one value per tier, fanned out to every env KV — nonprod envs share one vendor integration):

• EchoSignAppKey (Adobe Sign), SendGridApiKey, TMHPServerKey, zendeskSharedKey, HandShakingKey
• DevOpsTeamsWebhookURL, SystemHealthNotificationWebhookURLEligibility, SystemHealthNotificationWebhookURLImports

It refuses to propagate a PLACEHOLDER-* value and fails loudly if a secret is missing from the scaffold source (that would just re-create the silent 401), so you seed the scaffold KV first.

One-time scaffold seed (already done for nonprod 2026-07-07; once per tier). The real values are harvested from the legacy kv-eshars-qa — see the header of scripts/copy-kv-secrets.sh for the exact loop.

Deliberately NOT copied (do not add these to the copy list):

Secret	Why not	Correct source
EncryptionKey, EncryptionkeyByte	eSHARS crypto that must match data already encrypted in the DB — a blind copy corrupts	TF-generated tier key (synthetic env) or pinned via tier_encryption_key_override (migrated-data env)
masterPassword	NOT a crypto key — a universal auth backdoor. A non-empty value lets anyone log in as any user (the app’s auth paths accept it in place of the real password). Its SAFE state is ABSENT.	None — deliberately NOT provisioned (modules/key-vault-secrets no longer creates it; absent = bypass disabled). Never seed or copy a value. See eshars-security.
AzureFunctionsHostKey	Generated by the Function App resource	Read back from the resource
ImportApiUserPassword	App-internal cred paired with a SQL user	TF-generate + apply to SQL like SSRSReader (open follow-up)
Jwt--Secret	In use — the eshars-mobile-api HMAC-SHA256 signing key (KV Jwt--Secret → config Jwt:Secret; mobile-api Program.cs throws if unset). Not consumed by HISD, but not removable.	TF-generated (random_password.jwt_secret in modules/key-vault-secrets) — populated on apply. Do not hand-seed a dummy or remove it.

Terraform placeholders are protected with lifecycle { ignore_changes = [value] } — once the script writes a real value, future tofu apply runs leave it alone.

Other placeholders NOT seeded by copy-kv-secrets.sh

copy-kv-secrets.sh owns only the 8 shared-external vendor keys. A few other secrets are not copied — but most are now handled automatically (TF-generated on apply). Only the two below are genuine manual follow-ups.

Handled automatically — do NOT hand-seed:

• EncryptionKey / EncryptionkeyByte — TF-valued tier key (or pinned via tier_encryption_key_override for a migrated-data env). Guarded at apply time.
• Jwt--Secret — TF-generated (random_password.jwt_secret); the eshars-mobile-api signing key. If it’s still a placeholder the apply didn’t complete — re-run apply-env, don’t seed a dummy.
• masterPassword — deliberately NOT provisioned (a non-empty value is a universal auth backdoor; safe state is absent). Never set or copy a value. See eshars-security.

Genuine manual steps — set before declaring the env ready:

Secret	Impact if unset	Manual step
ImportApiUserPassword	Import API auth fails (imports only)	Generate a value, set the KV secret, create/align the matching SQL/API user (same pattern as SSRSReader)
AzureFunctionsHostKey	Functions-host-key callers fail (grid-settings/audit 401)	Read the host key back from the Function App post-deploy and set the KV secret

The stand-up is gated on the vendor keys + Jwt--Secret — see step 4c. The two manual secrets above are non-fatal to the env standing up (import/host-key features only) and are not gated.

4b. Create the SSRS report-execution login (SSRSReader)

SSRS report data sources connect to SQL as the low-priv SSRSReader login. The KV secrets (SSRSReaderUsername/SSRSReaderPassword) are TF-generated, but the SQL login itself is created by a post-apply script. Skip this and reports render with Login failed for user 'SSRSReader' → SSRS returns 500 (the other half of the 2026-07 dev SAForm break). This runs automatically in the stand-up workflow. SSRSReader is a plain SQL login (username/password), so the script connects as the server’s SQL admin — no Entra directory resolution, which is why the CI deploy SP runs it unattended (no managed identity, no Directory Readers grant). The admin credential is the same one used everywhere else in this runbook, read from two equivalent places:

• login — $TF_VAR_sql_server_admin_login if already exported (the workflow sets it, sourcing the SqlServerAdminLogin scaffold-KV secret — see prerequisite 6); otherwise the script reads that same SqlServerAdminLogin secret from the scaffold KV directly. Same value, two paths.
• password — SqlAdminPassword in the env KV (TF-generated random_password.sql_admin).

Both are present after apply (step 3) + the scaffold seed (prerequisite 5), so no manual step is needed. To run it by hand (same auth path):

./scripts/setup-db-ssrs-reader.sh --dry-run dev rg-eshars-data-nonprod-cus
./scripts/setup-db-ssrs-reader.sh dev rg-eshars-data-nonprod-cus

Also confirm the web app’s SSRSDomain app setting = the VM NetBIOS computer name (ssrs-{short}-{region}, e.g. ssrs-np-cus), NOT the Azure resource name vm-eshars-ssrs-{tier}-{region} — the wrong value gives a 401 from SSRS on the render call. (Sourced from report-server remote state computer_name output.)

4c. Gate: verify NO placeholders remain (mandatory)

The env is not ready until every Terraform-placeholder secret in its KV holds a real value. Run the gate — it fails (exit 1) and lists any secret still on PLACEHOLDER-*, the same way copy-kv-secrets.sh refuses placeholders for its 8:

./scripts/verify-env-secrets.sh dev

Do not proceed to smoke-testing the app while this fails. A leftover placeholder is a silent runtime failure (vendor 401, or — worse — unreadable encrypted data on a migrated env), not a deploy error, so this check is the only thing that turns “we documented the manual steps” into “the env actually has them”.

5. Apply Entra group grants to databases (MANUAL — human DBA, once per server)

Not run by the stand-up workflow — run this yourself

Unlike SSRSReader (step 4b, automated via SQL auth), this step is deliberately not in CI. It does CREATE USER … FROM EXTERNAL PROVIDER for the reader/writer/dba Entra groups, and Azure SQL permits creating Entra users only over an Entra-authenticated connection made by a member of the server’s Entra admin group (Only connections established with Active Directory accounts can create other Active Directory users). The CI deploy SP is not a resolvable group member, and granting it that ability is a privileged directory action we don’t automate (security review). So a human DBA (who is in the tier DBA group — their token carries the claim) runs it, once per server. It rarely changes and survives env rebuilds. (A future option is to front it as a self-service action in Port — the internal developer portal being introduced as the deployment front door, Phase 3 of the platform split — executed as the invoking DBA’s own Entra identity. Not built yet.)

The SQL server’s Entra admin (sg-eshars-db-nonprod-dba or sg-eshars-db-prod-dba) is set automatically by tofu apply. The database-level Entra group grants (reader, writer, dba) are not Terraform-managed — run the post-provisioning script as a DBA-group member:

# Dry run first
./scripts/setup-db-entra-groups.sh --dry-run dev rg-eshars-data-nonprod-cus

# Apply
./scripts/setup-db-entra-groups.sh dev rg-eshars-data-nonprod-cus

Env	Resource group
dev / qa / regression / staging	rg-eshars-data-nonprod-cus
uat / training / prod	rg-eshars-data-prod-cus

Groups applied for nonprod envs (per database-access-strategy):

Group	data	log	audit
sg-eshars-db-nonprod-reader	Read	Read	Read
sg-eshars-db-nonprod-writer	Read+Write+Exec	Read+Exec	Read+Exec
sg-eshars-db-nonprod-dba	db_owner	db_owner	db_owner

Requires go-sqlcmd with an active az login session (your account must be in the tier DBA group). Idempotent — safe to re-run.

6. App Service application settings (non-secret config)

Non-secret configuration (URLs, container names, flags) is Terraform-owned: workload-env/locals.tf (web_app_settings_defaults) derives sensible defaults from the env’s resource names and merges web_app_settings_override from the env’s .tfvars:

environments/dev.tfvars

web_app_settings_override = {
  "SSRSReportFolder" = "dev/Reports"
  # ... per-env additions/overrides
}

The merged map flows to the per-env app’s extra_app_settings — tofu apply pushes it. Secrets do not go here; they live in the env KV and reach the app via the greedy Key Vault config builder (see the architecture page for the full model, including why a KV secret of the same name overrides an app setting).

Only prod has a staging slot (deploy → warm → swap); env-specific settings are slot-sticky there so a swap moves code, not config.

7. Verify

# Resources in each tier RG (component-before-tier naming)
az resource list -g rg-eshars-security-nonprod-cus -o table
az resource list -g rg-eshars-data-nonprod-cus -o table
az resource list -g rg-eshars-storage-nonprod-cus -o table
az resource list -g rg-eshars-web-nonprod-cus -o table
az resource list -g rg-eshars-monitoring-nonprod-cus -o table

# The readiness gate again — must exit 0
./scripts/verify-env-secrets.sh dev

# DNS resolution from the spoke (run from a VM in the spoke VNet)
Resolve-DnsName eshars-primary.db.dev.eshars.internal   # PowerShell on the VM
dig eshars-primary.db.dev.eshars.internal               # Linux equivalent

# Web app reachable
curl -I https://app-eshars-dev-cus.azurewebsites.net/

Then deploy the application via the HISD repo’s tier-aware CD and smoke-test a report end-to-end (SAForm submit exercises Adobe Sign + SSRS + SSRSReader in one pass — the failure-signatures table maps every known symptom to its cause).

Tear-down

just init-env dev    # ensure backend is correct
tofu destroy -var-file=environments/dev.tfvars   # from infrastructure/workload-env

KV soft-delete retention is 7 days. If you re-create immediately, you’ll hit 409 Conflict on the env KV — purge it first:

az keyvault purge --name kv-eshars-dev-cus --location centralus

Open items / known gaps

• Placeholder automation — the two remaining manual secrets: ImportApiUserPassword (TF-generate + matching SQL user) and AzureFunctionsHostKey (post-deploy read-back). Until those land, step 4’s danger box + the 4c gate are the guardrail. (masterPassword and Jwt--Secret are already resolved — not-provisioned and TF-generated respectively, per eshars-ops #53 — no longer open items.)
• Post-apply scripts in CI — fold copy-kv-secrets.sh, setup-db-ssrs-reader.sh, and verify-env-secrets.sh into an apply-env pipeline (tracked).
• Dedicated low-priv SQL users for ESHARSREADONLY/ESHARSREPORTS — both connection-string secrets are TF-derived today (ESHARSREPORTS = the core connection string; ESHARSREADONLY = core with ApplicationIntent=ReadOnly). Swapping them to dedicated eshars_readonly/eshars_reports SQL users (the SSRSReader pattern) remains open.
• Teams webhook URL deprecation — Microsoft retired Office 365 connectors; use Power Automate Workflow URLs for TF_VAR_teams_webhook_url.
• Storage account name uniqueness — saesharsapp{env}{region} is globally unique across Azure. Confirm availability before stand-up: az storage account check-name --name <name>.
• GitHub environment bootstrap is manual — setup-github-environments.sh <env> (prerequisite 8) is a one-time user-PAT step outside CI, because the workload-env App token can’t create a repo environment (only manage vars/secrets inside one). Missing it 404s the apply. Folding it into an apply-env pipeline is blocked by the no-PAT-in-CI constraint (same reason it isn’t in Terraform).
• First-apply KV DNS race — even in-spoke, a brand-new env vault can 403 ForbiddenByConnection on the first secret writes: the module’s wait-for-pe-dns poll (getent) sees the fresh private A-record, but the tofu Go SDK can still hit a stale public answer cached before the PE existed. A re-apply clears it. A durable fix (poll via the SDK’s own resolver, or flush the cache pre-write) is open.

Cross-reference

• Why it works this way: Configuration & Secrets Architecture
• The tier-level chain this slots into: Deployment Order
• Scripts: eshars-ops/scripts/setup-github-environments.sh, copy-kv-secrets.sh, setup-db-ssrs-reader.sh, setup-db-entra-groups.sh, verify-env-secrets.sh
• KV secret module: eshars-ops/infrastructure/modules/key-vault-secrets/main.tf
• Workload-env root: eshars-ops/infrastructure/workload-env/