Hybrid Worker

The Hybrid Runbook Worker VM is required for long-running automation that exceeds Azure Automation’s 2-hour sandbox fair share limit. Currently used exclusively for the PII scrub pipeline.

Current State

Property	Value
VM Name	`vm-sql-access`
Resource Group	`vm-sql-access_group`
Location	East US 2
OS	Windows Server 2019 Datacenter
Size	Standard_B2ms (2 vCPU, 8 GB RAM)
Worker Group	`run-bacpacs-group`
Worker Type	HybridV2 (extension-based)
Automation Account	`aa-eshars-prod` (rg-eshars-prod, South Central US)
Private IP	10.3.2.4
Outbound IP	20.49.60.151
VNet	`vm-sql-access_group-vnet/default` (10.3.2.0/24)
SQL Firewall	`allow-vm-sql-access` on `db-eshars-developeraccess`

Migration Plan: Windows → Linux

Why Migrate

Reason	Detail
Cost	On-demand Linux cuts annual cost from ~$720 to under $60
Security	Windows Server 2019 is in extended support only (mainstream ended Jan 2024)
Compatibility	All runbooks use Az PowerShell + SqlServer module — fully cross-platform

Estimated Savings

Scenario	Monthly	Annual
Current: Windows 24/7	~$60	~$720
Linux 24/7	~$36	~$432
Linux + auto-shutdown (12h/day)	~$18	~$216
Linux + on-demand (recommended)	~$2–5	~$24–60

Phase 1: Provision Linux VM (parallel to existing)

# 1. Create Linux VM in the same VNet/subnet
az vm create \
  --resource-group vm-sql-access_group \
  --name vm-hybrid-worker-linux \
  --image Canonical:ubuntu-24_04-lts:server:latest \
  --size Standard_B2ms \
  --vnet-name vm-sql-access_group-vnet \
  --subnet default \
  --admin-username azureops \
  --generate-ssh-keys \
  --public-ip-address "" \
  --nsg ""

# 2. Install PowerShell 7
az vm run-command invoke \
  --resource-group vm-sql-access_group \
  --name vm-hybrid-worker-linux \
  --command-id RunShellScript \
  --scripts '
    apt-get update && apt-get install -y wget apt-transport-https software-properties-common
    wget -q "https://packages.microsoft.com/config/ubuntu/24.04/packages-microsoft-prod.deb"
    dpkg -i packages-microsoft-prod.deb && rm packages-microsoft-prod.deb
    apt-get update && apt-get install -y powershell
  '

# 3. Install Az + SqlServer modules
az vm run-command invoke \
  --resource-group vm-sql-access_group \
  --name vm-hybrid-worker-linux \
  --command-id RunShellScript \
  --scripts '
    pwsh -Command "
      Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
      Install-Module -Name Az.Accounts, Az.Automation, Az.Sql, Az.Storage, Az.Resources -Scope AllUsers -Force
      Install-Module -Name SqlServer -Scope AllUsers -Force
    "
  '

# 4. Register as Hybrid Worker
az vm extension set \
  --resource-group vm-sql-access_group \
  --vm-name vm-hybrid-worker-linux \
  --name HybridWorkerForLinux \
  --publisher Microsoft.Azure.Automation.HybridWorker \
  --version 1.1 \
  --settings '{"AutomationAccountURL": "<automation-account-url>"}'

# 5. Add to existing worker group
az automation hrwg hrw create \
  --automation-account-name aa-eshars-prod \
  --resource-group rg-eshars-prod \
  --hybrid-runbook-worker-group-name run-bacpacs-group \
  --name $(uuidgen) \
  --vm-resource-id "/subscriptions/9639039c-86c6-4859-b014-d28688863cd5/resourceGroups/vm-sql-access_group/providers/Microsoft.Compute/virtualMachines/vm-hybrid-worker-linux"

Phase 2: Update SQL Firewall

The new VM will have a different outbound IP. Update the firewall rule on db-eshars-developeraccess:

NEW_IP=$(az vm run-command invoke \
  --resource-group vm-sql-access_group \
  --name vm-hybrid-worker-linux \
  --command-id RunShellScript \
  --scripts 'curl -s https://api.ipify.org' \
  --query "value[0].message" -o tsv | grep -oP '\d+\.\d+\.\d+\.\d+')

az sql server firewall-rule update \
  --resource-group rg-eshars-nonprod \
  --server db-eshars-developeraccess \
  --name allow-vm-sql-access \
  --start-ip-address $NEW_IP \
  --end-ip-address $NEW_IP

Phase 3: Test

Run ar-eshars-database-scrub on run-bacpacs-group — the group now has two workers; Azure picks one. Verify it completes on the Linux worker. If it fails, the Windows worker is still available as fallback.

Phase 4: Decommission Windows VM

# Remove old worker from group
az automation hrwg hrw delete \
  --automation-account-name aa-eshars-prod \
  --resource-group rg-eshars-prod \
  --hybrid-runbook-worker-group-name run-bacpacs-group \
  --name a931a2ea-e21a-46d7-99d7-5546c1f597ee

# Deallocate — keep for 1 week as safety net before deleting
az vm deallocate --resource-group vm-sql-access_group --name vm-sql-access

Phase 5: Clean Up Firewall

If the new VM’s outbound IP differs from 20.49.60.151, remove the old rule from db-eshars-developeraccess.

Rollback

If issues arise during Phase 3:

Remove Linux worker from run-bacpacs-group
Windows VM is still registered and running
No changes needed to runbooks or firewall

On-Demand Execution

The biggest cost saving (~$660/year) comes from running the VM only during scrub jobs. Recommended approach: a wrapper runbook that starts the VM, runs the scrub, and stops the VM — all automatically.

# Runs on Azure sandbox (lightweight polling only — stays within fair share limit)
# The actual scrub runs on the Hybrid Worker

Connect-AzAccount -Identity

$vmRg    = 'vm-sql-access_group'
$vmName  = 'vm-hybrid-worker-linux'
$aaName  = 'aa-eshars-prod'
$aaRg    = 'rg-eshars-prod'

Write-Output "Starting $vmName..."
Start-AzVM -ResourceGroupName $vmRg -Name $vmName

# Wait for Hybrid Worker to reconnect after boot
Start-Sleep -Seconds 120

Write-Output "Starting scrub runbook on run-bacpacs-group..."
$job = Start-AzAutomationRunbook `
    -AutomationAccountName $aaName `
    -ResourceGroupName $aaRg `
    -Name 'ar-eshars-database-scrub' `
    -RunOn 'run-bacpacs-group'

do {
    Start-Sleep -Seconds 60
    $jobStatus = Get-AzAutomationJob `
        -AutomationAccountName $aaName `
        -ResourceGroupName $aaRg `
        -Id $job.JobId
    Write-Output "Scrub job status: $($jobStatus.Status)"
} while ($jobStatus.Status -notin 'Completed', 'Failed', 'Stopped', 'Suspended')

Write-Output "Stopping $vmName..."
Stop-AzVM -ResourceGroupName $vmRg -Name $vmName -Force

if ($jobStatus.Status -ne 'Completed') {
    throw "Scrub runbook finished with status: $($jobStatus.Status)"
}

Write-Output "Done — scrub completed, VM stopped."