Skip to content
eSHARS Operations

INFRA-0009: DNS Forwarder VM and P2S VPN Routing Model

Status

Accepted — supersedes the DNS/VPN sections of INFRA-0007 (2026-06-22)

Date

2026-06-22

Context

The rebuilt estate resolves private resources (SQL, Key Vault, Redis) over the P2S VPN through a dedicated DNS forwarder VM, not through the old ops VM. INFRA-0007 described the forwarder as vm-ops-nonprod at 10.1.0.12 with a single conditional forwarder — that is no longer how it works. This decision captures the current model and, importantly, the P2S client routing requirement that took a full debugging session to pin down.

Decision

The forwarder is a dedicated VM in its own Terraform root

control-plane-dns owns exactly one resource: a small Linux (dnsmasq) VM, pinned to the .4 address of the hub vms subnet.

TierForwarder VMStatic IPHub VNet
nonprodvm-eshars-dns-nonprod-cus10.20.4.4vnet-eshars-nonprod-cus (10.20.0.0/16)
prodvm-eshars-dns-prod-cus10.10.4.4(prod hub, 10.10.0.0/16)

The IP is cidrhost(<vms subnet cidr>, 4) and is stable across rebuilds, so the VPN gateway can push it to clients without a Terraform data dependency back to the gateway.

Why its own root: the forwarder is the operational churn in the hub (patched, rebuilt, re-onboarded by Defender). The hub VNet + VPN gateway + private DNS zones are static and stay in control-plane-core. Splitting the forwarder out lets it tear down and rebuild independently. Apply order: control-plane-core → control-plane-dns → workload-network → ….

dnsmasq forwards everything to Azure DNS

The cloud-init config is deliberately minimal:

no-resolv
server=168.63.129.16      # Azure platform DNS — recurses both public AND privatelink names
listen-address=127.0.0.1,<static-ip>
bind-interfaces
cache-size=0

Because it forwards all queries to 168.63.129.16 (which sees the VNet’s linked privatelink.* zones), the forwarder answers both private names (→ PE IPs) and public names (→ public recursion). There is no split or allow-list on the server side.

Client routing is split on the client, not the server

The client side is where private-vs-public is separated, and the two supported clients do it differently:

  • Linux (developer machines): scripts/dns-mode.sh enable writes a systemd-resolved drop-in that routes only a scoped suffix list (~database.windows.net, ~vault.azure.net, ~azurewebsites.net, ~redis.cache.windows.net, ~azconfig.io, ~servicebus.windows.net, and their privatelink.* forms) to the forwarder. Everything else resolves via the local public resolver. blob.core.windows.net is deliberately excluded — the Terraform state storage account is public + IP-allowlisted and must always resolve publicly, even when the VPN is down.
  • Windows (Azure VPN Client): the imported profile pins the forwarder as the DNS server. The client installs a catch-all NRPT rule (.) → all DNS, public included, goes to the forwarder.

The P2S routing requirement (hard-won)

A DNS server is only useful if the client has a route to it. On P2S, that route must be in the tunnel — pushed by the gateway or present in the client profile. If it is missing, every query to the forwarder times out, and the failure looks like a DNS problem when it is a routing problem.

Symptoms observed and what they actually mean:

SymptomReal cause
Linux resolves private names, internet works; Windows loses all DNS (incl. internet)The forwarder VM is down / not deployed. Linux scopes only private suffixes to the dead forwarder (internet still resolves locally); Windows’ catch-all sends everything to the dead forwarder → total failure.
nslookup <anything> <forwarder-ip> times out from Windows, but Linux on the same VPN worksWindows has no route to the hub VNet CIDR in its tunnel. Packets to the forwarder leave via the public default route and die. route print <hub-cidr> shows Active Routes: None.
Adding <dnssuffixes> to the Windows profile does not fix itSuffixes choose which resolver answers a name; they do not create the route to that resolver. Scoping is orthogonal to reachability.

Diagnostic order (use this, in order)

  1. Is the forwarder deployed and running? az vm list -d --query "[?contains(name,'dns')].{name:name,power:powerState,ips:privateIps}" A missing prod forwarder is the single most common cause of “Windows VPN broke.”
  2. Does the client have a route to the hub CIDR? Windows: route print 10.20.0.0 (nonprod) / 10.10.0.0 (prod) — expect an active route on the VPN interface, not None.
  3. Can the client reach the forwarder at all? nslookup microsoft.com <forwarder-ip> — a timeout with a present route means the forwarder is down; a timeout with no route means the route is the problem.

Resolution

  • Forwarder down → apply control-plane-dns for that tier (just apply-dns <tier>). The prod forwarder must exist for the prod VPN profile to resolve anything.
  • Missing route (immediate) → on the connected client, add a route over the tunnel interface (Windows: route add <hub-cidr> mask <mask> 0.0.0.0 IF <tunnel-ifIndex>).
  • Missing route (permanent) → the route belongs in the client profile, so every client gets it. The planned just vpn-publish templates azurevpnconfig.xml from the Terraform values already in state — vpn_client_address_pool, the hub VNet CIDR, legacy_vnet_cidrs, the forwarder IP, and the same suffix list dns-mode.sh uses — so the Windows profile cannot drift from what is actually deployed (<includeroutes> + <dnsservers> + <dnssuffixes>).

Consequences

  • A second DNS Private Resolver is not the fix for “VPN DNS broken.” The forwarder already recurses public names; a managed resolver placed in the same hub CIDR would be equally unreachable if the client has no route to that CIDR. Reachability is the lever, not server count.
  • The forwarder is single-AZ and a single VM — acceptable for a developer-access path, but it is a SPOF for VPN name resolution. If the VM is unhealthy, fall back to dns-mode.sh disable (Linux) and public DNS for anything public + IP.
  • dns-mode.sh disable (or never enabling it) is the correct mode during a teardown/rebuild, when the forwarder may not exist and tfstate must resolve publicly.