Skip to content
eSHARS Operations

TLS Certificate Renewal (App Gateway + Key Vault)

Overview

The *.myeshars.com wildcard certificate (GoDaddy, DV) is served from multiple places that renew differently:

WhereServesChain handling
App Service (wa-eshars-prod → www.myeshars.com)Uploaded certWindows schannel builds the chain automatically at handshake time
agw-eshars-np (rg-eshars-nonprod)nonprodreports, trainingreportsServes the Key Vault PFX verbatim — no chain building
appgw-eshars-nonprod-cus (rg-eshars-web-nonprod-cus)dev, qa, uat, regression, training, interview, npreportsSame — verbatim

Key Vault cert objects the gateways read (both via versionless secret references, so new versions are picked up automatically — eventually, see below):

  • kv-eshars-ag / wildcard-myeshars-com → agw-eshars-np
  • kv-eshars-scaf-nonprod / myeshars-wildcard → appgw-eshars-nonprod-cus

Renewal Procedure

1. Build a full-chain PFX

The PFX must contain, in order:

  1. Leaf (CN=*.myeshars.com)
  2. Intermediate (GoDaddy TLS Intermediate CA DV - R1v1)
  3. Cross-signed root (GoDaddy TLS Root CA - R1, issued by Go Daddy Root Certificate Authority - G2) — required for Chrome, which does not trust the standalone R1 root

The easiest source for the CA certs is whatever the working App Service endpoint currently serves (schannel emits the correct full chain):

Terminal window
# capture the served chain and split into one file per cert
echo | openssl s_client -connect www.myeshars.com:443 -servername www.myeshars.com -showcerts 2>/dev/null \
  | awk '/BEGIN CERTIFICATE/{n++; f=1} f{print > ("chain-" n ".pem")} /END CERTIFICATE/{f=0}'


# extract the private key + leaf from the renewed PFX, then rebuild with the chain
openssl pkcs12 -in renewed.pfx -nocerts -nodes -out key.pem -passin "pass:<pw>"
openssl pkcs12 -in renewed.pfx -clcerts -nokeys -out leaf.pem -passin "pass:<pw>"
cat leaf.pem chain-2.pem chain-3.pem chain-4.pem > fullchain.pem
openssl pkcs12 -export -inkey key.pem -in fullchain.pem -out renewed-fullchain.pfx -passout "pass:<pw>"
rm key.pem   # never leave the unprotected key on disk

2. Verify the PFX before importing

Terminal window
openssl pkcs12 -in renewed-fullchain.pfx -nokeys -passin "pass:<pw>" | grep -c "BEGIN CERTIFICATE"

Must be ≥ 3 (leaf + intermediate + cross-signed root). If it prints 1, stop — you have a leaf-only PFX.

3. Import into both Key Vaults

Import as a new version of the existing name — the gateways reference the versionless URI:

Terminal window
az keyvault certificate import --vault-name kv-eshars-ag           -n wildcard-myeshars-com -f renewed-fullchain.pfx --password "<pw>"
az keyvault certificate import --vault-name kv-eshars-scaf-nonprod -n myeshars-wildcard     -f renewed-fullchain.pfx --password "<pw>"

Confirm both imports actually landed (a failed import is easy to miss in a pasted block):

Terminal window
az keyvault certificate list-versions --vault-name kv-eshars-ag -n wildcard-myeshars-com \
  --query "[].{created:attributes.created, expires:attributes.expires}" -o table

4. Force the gateways to pick it up

App Gateway polls versionless Key Vault references roughly every 4 hours. There is no refresh API, and re-running az network application-gateway ssl-cert update with the same secret URI is a no-op (ARM reports Succeeded without re-fetching). The only reliable immediate refresh is a stop/start (~5–10 min outage per gateway):

Terminal window
az network application-gateway stop  -g rg-eshars-nonprod -n agw-eshars-np
az network application-gateway start -g rg-eshars-nonprod -n agw-eshars-np


az network application-gateway stop  -g rg-eshars-web-nonprod-cus -n appgw-eshars-nonprod-cus
az network application-gateway start -g rg-eshars-web-nonprod-cus -n appgw-eshars-nonprod-cus

If the outage is unacceptable, wait out the poll (≤ 4 h) instead.

5. Verify the served chain — not just the leaf

Terminal window
# must print 3+ (gateway sends the chain, minus optional trailing root)
echo | openssl s_client -connect nonprodreports.myeshars.com:443 \
  -servername nonprodreports.myeshars.com -showcerts 2>/dev/null | grep -c "BEGIN CERTIFICATE"


# expiry sanity check
echo | openssl s_client -connect nonprodreports.myeshars.com:443 \
  -servername nonprodreports.myeshars.com 2>/dev/null | openssl x509 -noout -enddate

Then confirm in Chrome specifically (see why below). Repeat for one host per gateway.

Troubleshooting / FAQ

Chrome says NET::ERR_CERT_AUTHORITY_INVALID but curl / PowerShell say the cert is fine

Different verifiers. curl on Windows and .NET use schannel, which auto-downloads missing intermediates (AIA fetching) and trusts the Microsoft root store. Chrome uses its own verifier and root store — it needs the intermediate and the R1 cross-signed by G2 bridge served on the wire. A chain that only passes schannel is not proven good; always verify with Chrome or check the served cert count.

I imported a new version but the gateway still serves the old cert

The ~4-hour poll hasn’t fired. ssl-cert update with the same URI won’t force it (no-op). Stop/start the gateway (step 4).

The browser still shows the error after the fix is verified on the wire

Chrome reuses live TLS connections. Hard-refresh (Ctrl+F5); if that fails, close all Chrome windows and reopen.

The cert expired but “we renewed it months ago”

Renewal ≠ deployment. The renewed cert can be live on App Service while the Key Vault copies the gateways read are still the old one. All three serving locations (App Service + both vault/gateway pairs) must be updated at each renewal.

Why does App Service work with a leaf-only PFX?

App Service frontends are Windows/IIS: schannel builds the full chain from the machine cert store at handshake time and serves it. Application Gateway v2 does no such completion — PFX in, identical bytes out.

History

  • 2026-07-27 — *.myeshars.com expired on both nonprod gateways (renewed cert existed since 2026-07-06 but was only deployed to App Service). Fix required two rounds: first import lacked the chain (leaf-only App Service export), causing Chrome ERR_CERT_AUTHORITY_INVALID even after the gateway refresh. Resolved by rebuilding the PFX with the full chain mirrored from www.myeshars.com. This runbook exists because of that incident.