Incident Response

Procedure for responding to production incidents.

Severity Levels

Level	Description	Response Time	Examples
SEV1	Complete outage	Immediate	Site down, data loss, prod DB unreachable
SEV2	Major degradation	30 minutes	Core feature broken, visit submission failing, login broken
SEV3	Minor impact	4 hours	Slow queries, intermittent errors
SEV4	Minimal impact	Next business day	Cosmetic issue, non-blocking warning

Initial Response

Step 1: Assess and Classify

1. Determine severity level
2. Identify affected users/districts
3. Identify affected components (HISD, mobile-api, mobile-ui, mgmt)
4. Check for ongoing deployments
5. Note incident start time

Step 2: Communicate

For SEV1/SEV2:

1. Notify on-call team via Teams
2. Create incident channel
3. Post initial status update

INCIDENT: [Brief description]
SEVERITY: [SEV1/2/3/4]
STATUS: Investigating
IMPACT: [Who is affected]
STARTED: [Time]

Step 3: Begin Investigation

1. Check Application Insights for error spikes
2. Review Azure Service Health for platform issues
3. Check recent deployments
4. Review infrastructure metrics

Investigation Checklists

Application Issues

• Check Application Insights exceptions
• Review recent code deployments
• Check application logs
• Verify configuration settings (Key Vault, app settings)

Database Issues

• Check Azure SQL metrics (DTU, connections)
• Review Query Performance Insight
• Check for blocking queries
• Verify connectivity via DNS alias

Infrastructure Issues

• Check Azure Service Health
• Review App Service metrics
• Check network connectivity / private endpoints
• Verify DNS resolution

Quick Diagnostics

# App Service status
az webapp show --name <app> --resource-group <rg> --query "state"

# Recent App Insights exceptions
az monitor app-insights query --app <app-insights-name> --resource-group <rg> \
    --analytics-query "exceptions | where timestamp > ago(1h) | summarize count() by type"

# SQL Server connectivity
az sql db show --resource-group <rg> --server <server> --name <db> --query "status"

Database Performance Diagnostics

When the database is slow or DTUs are spiking, run these queries from database/monitoring/ in order:

1. active-sessions.sql — What’s running right now? Shows CPU, elapsed time, current statement, blocking info
2. dtu-resource-breakdown.sql — Is it CPU, IO, or log writes driving the DTU spike?
3. session-details.sql — Who’s running the heavy query? Shows login name, host, program
4. query-store-top-consumers.sql — What’s been heaviest over the last 2 hours?
5. blocking-chains.sql — Are sessions blocking each other?

To kill a runaway session (safe for read-only queries):

KILL <session_id>;

To audit database access:

• entra-access-audit.sql — Lists all Entra ID users/groups and their database roles

Mitigation

• If app issue: rollback deployment (see Rollback Procedures)
• If DB issue: swap DNS alias (see Database Failover)
• If infrastructure issue: check Azure portal / az CLI

Resolution

Step 1: Implement Fix

1. Identify root cause
2. Develop fix or workaround
3. Test in lower environment if possible
4. Apply to production

Step 2: Verify

1. Confirm fix resolved issue
2. Monitor for 30 minutes after resolution
3. Check related functionality

Step 3: Communicate Resolution

INCIDENT RESOLVED
DURATION: [X hours/minutes]
ROOT CAUSE: [Brief explanation]
FIX: [What was done]
FOLLOWUP: [Any remaining actions]

Post-Incident

Within 24 Hours

1. Document timeline
2. Identify root cause
3. Create post-mortem document (see Incidents)

Within 1 Week

1. Conduct post-mortem meeting
2. Identify preventive measures
3. Create action items
4. Update runbooks if needed

Escalation Path

1. On-call Engineer
2. Engineering Lead
3. VP Engineering
4. Executive Team (SEV1 only)

Key Contacts

Role	Contact
On-call engineer	TBD
Team lead	TBD
DBA	TBD

---

See Incidents for all documented incidents and post-mortems.