Skip to content
eSHARS Operations

Job Alert Reference

Detailed reference for each alert type produced by the eSHARS job monitoring system.

STUCK JOBS DETECTED

Severity: Critical | Color: Red

Condition: A Quartz job has been in EXECUTING state for longer than the threshold (default: 60 minutes).

Query: Checks QRTZ_FIRED_TRIGGERS where STATE = 'EXECUTING' and FIRED_TIME is older than the threshold.

Card contains:

  • Job name
  • How many minutes it has been running
  • Instance name (which scheduler host)

Recommended response:

  1. Check if the job is legitimately long-running (e.g., large import batch)
  2. Look for SQL blocking via database/monitoring/blocking-chains.sql
  3. Check external dependencies (TMHP, SFTP, SendGrid)
  4. If genuinely hung, restart the Job Scheduler App Service
  5. See runbook for full procedure

DEAD SCHEDULER DETECTED

Severity: Critical | Color: Red

Condition: A Quartz scheduler instance hasn’t checked in within 2x its configured check-in interval.

Query: Checks QRTZ_SCHEDULER_STATE comparing LAST_CHECKIN_TIME against the current time.

Card contains:

  • Instance name
  • Scheduler name

Recommended response:

  1. Check the App Service hosting the Job Scheduler in Azure portal
  2. Look for crash logs in Application Insights
  3. Restart the App Service
  4. See runbook for full procedure

Job Errors

Severity: High | Color: Red

Condition: Any ERROR or FATAL level log entries in the job execution logs within the lookback window (default: 30 minutes).

Query: Checks JobExecutionLogs joined with JobExecutions for recent error-level entries.

Card contains:

  • Job name (up to 10 most recent)
  • Truncated error message

Recommended response:

  1. Open eshars-mgmt → Jobs → Errors for the environment
  2. Review full error message and stack trace
  3. See runbook for common causes

Repeated Job Failures

Severity: High | Color: Orange

Condition: A job has failed on its last 3+ consecutive executions (all recent runs have errors).

Query: Checks the last N completed executions per job and cross-references with error logs.

Card contains:

  • Job name
  • Number of consecutive failures

Recommended response:

  1. This indicates a persistent problem, not a transient error
  2. Review error details for the last several executions
  3. Likely needs a code fix, config change, or external dependency resolution
  4. See runbook for procedure

Jobs Not Running

Severity: Medium | Color: Orange

Condition: A job that has historically run hasn’t executed in the last 24 hours.

Query: Checks JobExecutions for the most recent Status = 1 (started) entry per job.

Card contains:

  • Job name
  • Hours since last run

Recommended response:

  1. Check if the job’s Quartz trigger is in PAUSED or ERROR state
  2. Verify the scheduler is alive
  3. Check if the job was intentionally disabled
  4. See runbook for procedure

High Failure Rate

Severity: Medium | Color: Orange

Condition: A job’s failure rate exceeds the threshold (default: 25%) over the last 24 hours.

Query: Counts completed executions and error-flagged executions per job, calculates percentage.

Card contains:

  • Job name
  • Failure rate percentage
  • Failed/total run counts

Recommended response:

  1. Low total runs with a high percentage may be noise (e.g., 1/2 = 50%)
  2. High total runs with a high percentage indicates a real problem
  3. Cross-reference with recent deployments
  4. See runbook for procedure