Skip to content
eSHARS Operations

eSHARS Application Infrastructure — OpenTofu

Overview

The myeshars.com application infrastructure is managed as OpenTofu (tofu) roots in the eshars-ops repo under infrastructure/. It provisions the Azure resources that support the HISD web portal (myeshars.com), the function app (API), the management portal, and the supporting data/identity layer.

The single infrastructure/eshars/ root described in older docs no longer exists. Application resources are now split into tier-based roots:

  • workload-core/ — tier-shared platform: App Service Plan, the tier web app + function app, Redis, the App Gateway, the tier-shared Key Vault, the staging storage account, and the tier Automation Account. Applied once per tier (nonprod, prod).
  • workload-env/ — per-environment resources: SQL Server, Key Vault, storage account, Application Insights, private DNS, monitoring, and the per-environment deployment slot on the tier app. Applied once per environment (dev, qa, regression, training, uat, prod).
  • workload-network/ — the spoke VNet and subnets (apps, private endpoints, App Gateway). See Workload Network.

Networking peering, the hub VNet, and the core private DNS zones (privatelink.database.windows.net, privatelink.blob.core.windows.net, privatelink.vaultcore.azure.net) are owned by the control-plane roots — see Landing Zone Architecture.

Infrastructure Roots

infrastructure/
├── control-plane-core/       # Hub VNet, peering, core SQL private DNS zone
├── control-plane-dns/        # DNS forwarder / resolver config
├── control-plane-dns-zones/  # Global private DNS zones (blob, vaultcore)
├── workload-network/         # Spoke VNet + subnets (apps, PE, appgw)
├── workload-core/            # Tier-shared: ASP, web/func app, Redis, App GW, AA
├── workload-env/             # Per-env: SQL, KV, storage, App Insights, slots
├── vms/                      # job-scheduler, report-server, SFTP, ops VMs
├── modules/                  # Reusable Terraform modules
├── scalr/                    # Scalr / runstate config
├── port/                     # Port resource catalog
└── docs/                     # Architecture documentation

Architecture

┌──────────────────────────────────────────────────────────────┐
│  control-plane-core / -dns-zones                              │
│  Hub VNet + peering                                           │
│  Core private DNS zones:                                      │
│    privatelink.database.windows.net                          │
│    privatelink.blob.core.windows.net                         │
│    privatelink.vaultcore.azure.net                           │
└──────────────────────────────────────────────────────────────┘
                              │ peering
                              ▼
┌──────────────────────────────────────────────────────────────┐
│  workload-network/  (per tier)                                │
│  Spoke VNet: vnet-eshars-{tier}-cus                           │
│  ├── snet-apps-eshars-{tier}-cus    (App Service integration) │
│  ├── snet-pe-eshars-{tier}-cus      (private endpoints)       │
│  └── snet-appgw-eshars-{tier}-cus   (App Gateway)             │
└──────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┴───────────────────────┐
        ▼                                              ▼
┌────────────────────────────┐         ┌────────────────────────────┐
│ workload-core/  (per tier)  │         │ workload-env/  (per env)    │
│                             │         │                             │
│ asp-eshars-{tier}-cus       │         │ sql-eshars-{env}-cus        │
│ app-eshars-{tier}-cus  ◄────┼─slots───┤   (per-env SQL Server)      │
│ func-eshars-{tier}-cus ◄────┼─slots───┤ kv-eshars-{env}-cus         │
│ redis-eshars-{tier}-cus     │         │ saesharsapp{env}cus         │
│ appgw-eshars-{tier}-cus     │         │ appi-eshars-{env}-cus       │
│ kv-eshars-ts-{tier}-cus     │         │ mon-eshars-{env}-cus        │
│ saeshars{tier}cus (staging) │         │ db.{env}.eshars.internal    │
│ aa-eshars-{tier}-cus        │         │ SQL + blob private endpoints│
└────────────────────────────┘         └────────────────────────────┘

Each environment runs as a deployment slot on the tier-shared web/function app, not as a standalone app. dev is the production slot of the nonprod-tier app and prod is the production slot of the prod-tier app; the remaining envs (qa, regression, training, uat) are named slots.

Roots & Modules

workload-core (tier-shared)

Applied once per tier (nonprod, prod). Modules:

ModuleResourceNotes
app-service-planasp-eshars-{tier}-cusS1 (nonprod), P1v3 (prod)
web-appapp-eshars-{tier}-cusWindows, .NET 4.x, VNet-integrated, /health probe
function-appfunc-eshars-{tier}-cusWindows, uses the staging SA for storage
redisredis-eshars-{tier}-cusStandard; capacity 0 (nonprod) / 1 (prod)
app-gatewayappgw-eshars-{tier}-cusConditional (deploy_app_gateway); WAF + SSL termination; backends = web app + SSRS VM
automationaa-eshars-{tier}-cusBACPAC-restore runbooks
rbac—Deploy-identity grants (ops VM MI, legacy GitHub SP, AA MI)

It also creates the 8 tier resource groups (rg-eshars-{component}-{tier}-cus: security, data, storage, web, monitoring, compute, ops, backup), the staging storage account (saeshars{tier}cus), and the tier-shared Key Vault (kv-eshars-ts-{tier}-cus, which holds the *.myeshars.com wildcard cert for App Gateway SSL termination).

workload-env (per environment)

Applied once per environment. Modules:

ModuleResourceNotes
sql-serversql-eshars-{env}-cusv12.0, TLS 1.2, Entra admin via tier DBA group
sql-databasecore / log / audit DBsServer only — schema owned by HISD DbUpgrader
key-vaultkv-eshars-{env}-cusRBAC, private endpoint, purge-free (clean teardowns)
key-vault-secrets—Connection strings, encryption keys, SSRS creds, App Insights key
storagesaesharsapp{env}cusCMK-encrypted (per-env key in scaffold KV)
app-insightsappi-eshars-{env}-cusPer-env Application Insights
private-dnsdb.{env}.eshars.internalDB alias zone (see below)
monitoringmon-eshars-{env}-cusAutomation runbook + action group for Quartz job health
rbac—Slot / SSRS / job-scheduler / ops MI → KV Secrets User

It also creates the per-env deployment slot on the tier web/function app (skipped for the production-slot envs), the SQL and blob private endpoints in the PE subnet (registered in the hub-managed privatelink.* zones), a CanNotDelete lock + prevent_destroy on the prod SQL server, and syncs per-env deploy variables/secrets to the HISD GitHub repo environment.

private-dns (DB alias layer)

DNS abstraction for zero-downtime database switching:

  • Zone: db.{env}.eshars.internal (per environment)
  • CNAME records: eshars-primary, eshars-reporting → SQL server FQDN
  • CNAME targets are managed operationally (swap scripts); Terraform uses lifecycle { ignore_changes } and does not manage the targets.

Apps connect to eshars-primary.db.{env}.eshars.internal, which resolves via the private endpoint to the SQL private IP — so the server can change without a config redeploy.

monitoring (Quartz job health)

Per-env Azure Automation account + scheduled PowerShell runbook that watches for stuck jobs, dead schedulers, recent ERROR/FATAL log entries, consecutive failures, stale jobs (24h+), and high failure rates. Alerts go to a Teams webhook and an email action group.

Naming Convention

All resources follow Azure CAF naming. eshars is the product; the customer/district (hisd) is not part of resource names. Region is a 2–3 char code (cus = Central US). See INFRA-0005: Resource Naming Convention for the full rationale.

  • Tier-shared (workload-core): {type}-eshars-{tier}-cus — e.g. asp-eshars-nonprod-cus, app-eshars-prod-cus, redis-eshars-prod-cus.
  • Per-environment (workload-env): {type}-eshars-{env}-cus — e.g. sql-eshars-regression-cus, kv-eshars-dev-cus, appi-eshars-qa-cus.
PrefixResource
rg-Resource Group
asp-App Service Plan
app-Web App
func-Function App
kv-Key Vault
sql-SQL Server
redis-Redis Cache
appgw-Application Gateway
appi-Application Insights
aa-Automation Account
pe-Private Endpoint

Storage accounts drop dashes for the 24-char cap: saeshars{tier}cus (staging, tier-shared) and saesharsapp{env}cus (per-env; the app infix avoids a global-name collision with the staging SA at environment=prod). The tier-shared Key Vault carries a ts (tier-shared) infix — kv-eshars-ts-{tier}-cus — because prod is both a tier name and an env name, so kv-eshars-prod-cus would otherwise collide with the prod-env KV.

Environments

Six environments across two tiers. The tier (local.infra_tier = prod | nonprod) drives the shared platform a given env runs on.

EnvironmentTierAlways-OnSlot
devnonprodnoproduction slot (nonprod app)
qanonprodnonamed slot
regressionnonprodnonamed slot
trainingprodyesnamed slot
uatprodyesnamed slot
prodprodyesproduction slot (prod app)

Usage

Run from the repo root via the justfile. wc = workload-core (takes a tier), env = workload-env (takes an environment), wn = workload-network.

Terminal window
# Tier-shared platform (per tier: nonprod | prod)
just init-wc nonprod
just plan-wc nonprod
just apply-wc nonprod


# Per-environment resources (dev | qa | regression | training | uat | prod)
just init-env dev
just plan-env dev
just apply-env dev


# Spoke network (per tier)
just apply-wn nonprod

Each recipe wraps tofu with the matching environments/{name}.backend.hcl and environments/{name}.tfvars.

Sensitive Variables

Secrets live in Key Vault, never in *.tfvars. The few values Terraform itself needs are passed as TF_VAR_ environment variables (set in CI/CD or the shell):

VariableEnv var
sql_server_admin_loginTF_VAR_sql_server_admin_login
teams_webhook_url (optional)TF_VAR_teams_webhook_url

Per-env SQL admin passwords are generated by Terraform (random_password) and written into the env Key Vault and the GitHub Actions secret store. Per-env storage CMK keys and the *.myeshars.com wildcard cert live in the purge-protected scaffold KV (kv-eshars-scaf-{tier}), which is created out-of-band by setup-terraform-backends.sh and survives teardowns.

State

Backend: azurerm, per tier. State lives in the scaffold storage account for that tier.

TierStorage accountResource groupContainer
nonprodsattfnonprodcusrg-eshars-scaffold-nonprod-custfstate
prodsattfprodcusrg-eshars-scaffold-prod-custfstate

State keys are per root/env, e.g. workload-core-nonprod.tfstate, workload-env-dev.tfstate, workload-network-prod.tfstate. Auth is use_azuread_auth = true.

Cross-Root Dependencies

Roots read each other’s outputs via terraform_remote_state. Most reads are graceful — they default to null until the dependency is applied, so bootstrap doesn’t deadlock; re-apply the dependent root once the source exists.

workload-env reads:

  • workload-core (per tier) — tier web/function app IDs and hostnames, to create the per-env deployment slot and derive service URLs.
  • report-server (per tier, graceful) — SSRS URL, VM name, and admin creds, for app settings and KV secrets.
  • job-scheduler (per tier, graceful) — VM principal ID, to grant the job-scheduler MI KV Secrets User.

workload-core reads:

  • vms/ops (graceful) — ops VM principal ID, to grant Website Contributor on the tier web app (the CD runner’s deploy rights).
  • report-server (when deploy_app_gateway = true, graceful) — SSRS private IP + URL, to register the SSRS backend in App Gateway.

For the full apply order, see Deployment Order.