Runbooks
Reusable operational procedures for managing eSHARS in production.
Purpose
Runbooks provide step-by-step procedures for:
- Incident response
- Routine maintenance
- Emergency procedures
- Disaster recovery
Runbook Index
| Runbook | Description | When to Use |
|---|---|---|
| Incident Response | Handle production incidents | Production issue detected |
| Backup & Restore | Database backup procedures | Scheduled or emergency |
| Database Failover | SQL Server failover/swap procedure | Planned maintenance or unhealthy primary |
| Database Maintenance | Routine DB maintenance | Weekly/monthly |
| Pipeline Failure | Troubleshoot CI/CD pipeline failures | Build or release pipeline fails |
| Rollback Procedures | Roll back App Service or database deployments | Failed deployment |
| Release Process | Production deployment | Each release |
| Report (SSRS) Deployment | Publish report RDLs + data sources to the SSRS server | Release with new/changed reports, or report publish recovery |
| Activity Log Audit | Query Azure Activity Log for changes | Investigating who changed what |
| Job Monitoring & Alerts | Respond to automated job monitoring alerts | Job alert triggered |
| TMHP Response Staging Alerts | Investigate TMHP EDI response-staging alerts (270/271 batch flow) | ResponseStaging file/age threshold alert fires |
| Jira Automation Rules | HM project Jira automation rules | Managing ticket automation |
| VM Troubleshooting | Azure VM diagnostics and fixes | VM issues (WAD, connectivity) |
| AthenaPractice Datacenter Migration | Datacenter migration for AthenaPractice integration | IP/URL/cert changes |
| Job Scheduler: Pause & Resume | Pause/resume scheduled jobs (BillTransaction, Eligibility, etc.) | Vendor maintenance, emergency stop |
| Device Compliance Audit | Audit non-compliant Intune devices | Monthly or onboarding/offboarding |
| Key Vault: Move Between Resource Groups | Safely move a Key Vault to a different RG without breaking web app access | Infrastructure reorganisation |
| TLS Certificate Renewal | Renew the *.myeshars.com wildcard across App Service + both App Gateways (full-chain PFX, Key Vault import, gateway refresh) | Cert renewal (next: Jan 2027) or browser cert errors |
Quick Reference
Production Issue?
- Start with Incident Response
- Assess severity
- Follow escalation procedures
- Document in incident log
Scheduled Maintenance?
- Notify stakeholders
- Follow specific runbook
- Verify completion
- Update maintenance log
Contact Information
| Role | Contact |
|---|---|
| On-call Engineer | Check PagerDuty |
| Engineering Lead | [internal contact] |
| DevOps | [internal contact] |
| Support | support@myeshars.com |
Maintenance Windows
| Window | Time (CT) | Purpose |
|---|---|---|
| Weekly | Sunday 2-4 AM | Routine maintenance |
| Monthly | First Sunday 12-4 AM | Extended maintenance |