Job Monitoring & Alerts

Procedures for responding to automated job monitoring alerts. Alerts are sent to Teams by the eSHARS-Job-Monitor Azure Automation Runbook running every 15 minutes.

Alert Reference

Alert	Severity	Color	Condition	Immediate Action
STUCK JOBS	Critical	Red	Job executing > 60 min	Check if legitimately long-running or hung
DEAD SCHEDULER	Critical	Red	Scheduler instance not checking in	Restart App Service / investigate host
Job Errors	High	Red	ERROR/FATAL in last 30 min	Review error in eshars-mgmt or ESHARSLOGS
Repeated Failures	High	Orange	3+ consecutive failures for same job	Investigate root cause, may need code fix
Jobs Not Running	Medium	Orange	Job hasn’t run in 24+ hours	Check if trigger is disabled or scheduler is down
High Failure Rate	Medium	Orange	>25% failure rate over 24h	Review error patterns, may be transient

Investigation Steps

Stuck Job

1. Open eshars-mgmt → Jobs → Running for the environment
2. Identify the stuck job and note how long it’s been running
3. Run database/monitoring/job-stuck.sql against the ESHARS database for details
4. Check if the job is waiting on an external dependency (TMHP, SFTP, etc.)
5. Check database/monitoring/blocking-chains.sql for SQL-level blocking
6. If the job is genuinely hung:
   • Check if it’s safe to kill (import jobs may leave partial data)
   • Restart the App Service hosting the Job Scheduler if needed
   • Document in the log below

Dead Scheduler

1. Check Azure portal → App Services for the Job Scheduler host
2. Verify the App Service is running and healthy
3. Check Application Insights for any crash logs
4. Run database/monitoring/job-scheduler-health.sql against ESHARS
5. If the scheduler is down:
   • Restart the App Service
   • Monitor for 5 minutes to confirm check-ins resume
   • If it crashes again, check recent deployments or config changes

Job Errors

1. Note which job(s) are failing from the Teams alert
2. Open eshars-mgmt → Jobs → Errors for the environment
3. Click into the execution to see full error + stack trace
4. Run database/monitoring/job-errors-recent.sql for additional context
5. Common causes:
   • Database connectivity: Check Azure SQL health, DNS resolution
   • External service down: TMHP, SendGrid, SFTP — check service status
   • Data issues: Bad input data causing validation failures
   • Timeout: Long-running queries — check database/monitoring/slow-queries.sql

Consecutive Failures

1. This indicates a persistent problem, not a transient error
2. Run database/monitoring/job-consecutive-failures.sql to see affected jobs
3. Review the error details for the last 3+ executions in eshars-mgmt
4. If the same error repeats: likely needs a code fix or config change
5. If different errors each time: may be an infrastructure issue

Jobs Not Running

1. Run database/monitoring/job-not-run-recently.sql to see which jobs are stale
2. Check if the job’s trigger is still active in the Quartz tables:

   SELECT * FROM QRTZ_TRIGGERS WHERE JOB_NAME = 'TheJobName'

3. If trigger state is PAUSED — someone may have intentionally paused it
4. If trigger state is ERROR — the trigger itself failed, may need to be reset
5. If the scheduler is running but the job isn’t firing, check misfire settings

High Failure Rate

1. Run database/monitoring/job-failure-rate.sql to see rates per job
2. A high rate with low total runs may be noise (e.g., 1/2 = 50%)
3. A high rate with many runs indicates a real problem
4. Cross-reference with recent deployments — did a code change break something?

Adjusting Thresholds

Thresholds are configured in the Terraform module at infrastructure/modules/monitoring/variables.tf:

Parameter	Default	Description
schedule_frequency_minutes	15	How often the monitor runs
stuck_threshold_minutes	60	Minutes before a job is “stuck”
error_lookback_minutes	30	Window for recent error detection
failure_rate_threshold	25	Percentage threshold for high failure rate

To change: the monitoring module (infrastructure/modules/monitoring/) is called per environment from the workload-env root — update the module call in infrastructure/workload-env/main.tf, then just plan-env <env> / just apply-env <env>. (The single-root infrastructure/main.tf referenced by older versions of this page no longer exists.)

Silencing Alerts

If a known issue is being worked on and alerts are noisy:

1. Temporarily: Disable the Automation schedule in Azure portal → Automation Accounts → Schedules
2. Per-job: No per-job suppression yet — file an enhancement request if needed
3. Re-enable: Don’t forget to re-enable after the issue is resolved

Escalation

Severity	Response Time	Escalate To
Critical	15 minutes	On-call engineer, then team lead
High	1 hour	Assigned developer
Medium	Next business day	Team standup

Change Log

Date	Change	By
	Initial setup