GitHub Actions Runner on the Ops VM

The per-tier ops VM (vm-eshars-ops-{tier}-cus) hosts the self-hosted GitHub Actions runners that run CD deploys. Runners are org-level (shared across HISD, eshars-ops, and other AuthenticaTech repos), installed automatically when the VM is stood up — no manual RDP step.

How it works

When the ops VM boots, its CustomScriptExtension runs two phases in order:

1. install-ops-modules.ps1 — prerequisites: PowerShell 7, Az modules, git-bash, Azure CLI, and jq (the CD composite actions and deploy scripts need bash + az + jq on PATH). This phase is fatal — if it fails the VM apply fails.
2. install-github-runner.ps1 — installs N runner services, self-registering via a GitHub App whose credentials are read from the scaffold Key Vault using the VM’s managed identity.

ops VM boots
  -> CSE phase 1: install-ops-modules.ps1   (pwsh7 + Az + git-bash + az CLI + jq)
  -> CSE phase 2: install-github-runner.ps1
       -> az login --identity                    (VM managed identity)
       -> read GitHub App id/installation/key     (scaffold KV: kv-eshars-scaf-{tier})
       -> App JWT  -> installation token          (api.github.com)
       -> installation token -> registration token
       -> config.cmd --runasservice x N           (joins runner group "eshars-deploy")

No long-lived registration token is stored anywhere — it is minted fresh at boot and expires in ~1 hour. The only durable secret is the GitHub App private key, which lives in the scaffold Key Vault.

Runner version is pinned

install-github-runner.ps1 pins RunnerVersion = 2.335.1. The runner must be ≥ 2.327.1: azure/webapps-deploy@v3 (and other actions) declare using: node24, which older runners fail to load with “‘using: node24’ is not supported”. Don’t fetch latest at boot — pin a known-good version.

Labels and isolation

Runners are labeled eshars,{tier},windows. The tier label is the deploy isolation boundary:

• nonprod deploys target runs-on: [self-hosted, eshars, nonprod]
• prod deploys target runs-on: [self-hosted, eshars, prod]

This guarantees a nonprod workflow can never land a job on the prod ops VM (which holds prod Key Vault / SQL access).

The label is eshars, not eshars-ops

eshars and eshars-ops are two distinct labels. A workflow targeting [self-hosted, eshars, nonprod] will never match a runner advertising eshars-ops — the job queues forever with no runner. runner_labels in vms/ops tfvars must be eshars,{tier},windows and must match the workflow’s runs-on exactly.

Which repos may use the runners is controlled by the runner group (eshars-deploy) repository-access setting in the org — not by labels.

---

One-time setup (manual — not Terraform)

These cannot be safely automated and must be done once per org / per tier.

1. Create the GitHub App (once, org-wide)

1. Org → Settings → Developer settings → GitHub Apps → New GitHub App.
2. Permissions: Organization → Self-hosted runners: Read & write. Nothing else.
3. Create, then Install the App on the AuthenticaTech org.
4. Record: App ID, Installation ID (from the install URL), and generate + download a private key (PEM).

Why a GitHub App and not a PAT: no user account to tie it to, independent rotation, survives staff changes, and minimal scope. Right for an org-level production runner.

2. Create the runner group (once, org-wide)

Org → Settings → Actions → Runner groups → New runner group:

• Name: eshars-deploy (must match runner_group in Terraform)
• Repository access: select HISD, eshars-ops, and any other repos that should be allowed to run deploys on these runners.

The bootstrap script does not create this group — it fails with a clear message if the group is missing, rather than silently falling back to the all-repos default group.

3. Store the App secrets in each tier scaffold Key Vault (once per tier)

The VM MI reads these three secrets from kv-eshars-scaf-{tier} (the scaffold vault — author-once material that lives outside the teardown/rebuild cycle, so a tier rebuild never has to re-import them). Set them out-of-band (you need Key Vault Secrets Officer on the vault):

# nonprod (repeat with kv-eshars-scaf-prod for prod)
az keyvault secret set --vault-name kv-eshars-scaf-nonprod \
  --name gh-runner-app-id          --value "<APP_ID>"
az keyvault secret set --vault-name kv-eshars-scaf-nonprod \
  --name gh-runner-installation-id --value "<INSTALLATION_ID>"
az keyvault secret set --vault-name kv-eshars-scaf-nonprod \
  --name gh-runner-private-key     --file ./eshars-runner.private-key.pem

The ops VM MI is granted Key Vault Secrets User on the scaffold KV by Terraform (vms/ops, grant key ops-mi-scaffold-kv-secrets), so no manual access-policy step is needed.

Scaffold KV, not the tier KV

Runner secrets live in the scaffold vault on purpose. The scaffold KV is the only purge-protected vault and sits outside the per-tier teardown cycle, so author-once secrets (runner App key, the wildcard TLS cert) survive rebuilds. The per-tier KV (kv-eshars-ts-{tier}-{region}) is created purge-free and is not where runner secrets go. See INFRA-0009 and the scaffold-KV rationale in the ops repo.

---

Standing it up

just apply-wc {tier}     # uploads the runner + bootstrap scripts to the scripts container
just apply-ops {tier}    # builds the VM; CSE installs prereqs + registers the runners

(apply-wc first — it publishes install-github-runner.ps1 / install-ops-modules.ps1 to the tier staging account’s scripts container, which the ops VM CSE then downloads.)

Verifying

1. On the VM (RDP or Run Command): Get-Service actions.runner.* → all Running.
2. Confirm the toolchain is on PATH: pwsh -v (7.x), bash --version, az version, jq --version.
3. Org → Settings → Actions → Runners → group eshars-deploy: the runners show Idle, labels eshars,{tier},windows.
4. Reboot the VM → runners reconnect automatically (installed as services).
5. A workflow with runs-on: [self-hosted, eshars, {tier}] picks up a job.

Notes / troubleshooting

• Job queues forever, no runner picks it up → label mismatch. The workflow’s runs-on must be [self-hosted, eshars, {tier}] and the runner must advertise eshars,{tier},windows. eshars-ops is a different label — see the caution above.
• bash / jq not found in a CD step → git-bash keeps Git\bin off the machine PATH by default, and jq is added under Git\usr\bin. install-ops-modules.ps1 adds both to the machine PATH, but runner services cache the environment at start — restart the runner services (or reboot) after a PATH change.
• 'using: node24' is not supported → the runner is older than 2.327.1. The pin is 2.335.1; an in-place upgrade (config.cmd remove → re-extract → re-register) fixes a VM that was built on an older pin.
• Runner count is runner_count (default 2) in vms/ops — concurrent jobs per VM. Raise it if deploys queue.
• Rebuild safety: config.cmd --replace lets a rebuilt VM reclaim the same runner names cleanly, so a VM rebuild does not orphan registrations.
• Egress: an explicit NSG rule allows outbound 443 to the Internet (GitHub). Without it, a tightened deny-all-outbound NSG would silently break registration.
• Group missing → the script throws Runner group 'eshars-deploy' not found. Create the group (step 2) and re-run.
• Secrets missing → the script throws a missing-secret error naming the KV. Confirm step 3 ran against the correct tier scaffold vault.