Pause & Resume Quartz Scheduled Jobs

Created: 2026-03-26 Category: Operational Database: db-eshars-prod (ESHARS connection string) Scheduler: Quartz.NET with AdoJobStore (SQL persistence)

When to Use

• Vendor system maintenance (e.g., AthenaPractice data center migration)
• Infrastructure changes that could affect job connectivity
• Emergency: jobs sending data to wrong endpoint or causing errors
• Any situation where jobs must be temporarily stopped without restarting the Job Scheduler Windows service

Background

The eSHARS Job Scheduler uses Quartz.NET with SQL-backed persistence (QRTZ_* tables in the eSHARS database). BillTransactionJobs are not registered in code — they are created via the eSHARS admin UI and stored entirely in the Quartz database tables. The scheduler honors the TRIGGER_STATE column in QRTZ_TRIGGERS:

• WAITING — trigger is active and will fire at next scheduled time
• PAUSED — trigger is suspended and will not fire
• ACQUIRED — scheduler has picked up the trigger (about to execute)
• EXECUTING — job is currently running

Source: Eshars.JobScheduler/JobService.cs line 368 confirms BillTransactionJob is excluded from code-based registration. Config at Eshars.JobScheduler/App.config line 25 confirms AdoJobStore.

Important: Service Restart and Resume Behavior

• Service restart: PAUSED triggers survive a Job Scheduler service restart — AdoJobStore persists state to SQL. The startup path (Start() > RegisterJobTypes()) does not call ResumeAll(), so paused triggers stay paused.
• Service “Continue” (unpause): The Topshelf Continue() handler calls _scheduler.ResumeAll(), which resumes all paused triggers — including ones you intentionally paused. Do not use the Windows Service “Continue” action during a maintenance window.

---

Procedure: Pause BillTransactionJobs

Step 1: Check Current State

-- Run against db-eshars-prod
-- Confirms which triggers exist and that they are in WAITING state before you pause
SELECT
    t.TRIGGER_NAME,
    t.TRIGGER_GROUP,
    t.TRIGGER_STATE,
    ct.CRON_EXPRESSION,
    ct.TIME_ZONE_ID
FROM QRTZ_TRIGGERS t
LEFT JOIN QRTZ_CRON_TRIGGERS ct
    ON t.SCHED_NAME = ct.SCHED_NAME
    AND t.TRIGGER_NAME = ct.TRIGGER_NAME
    AND t.TRIGGER_GROUP = ct.TRIGGER_GROUP
WHERE t.TRIGGER_NAME LIKE '%BillTransaction%';

Note the number of rows and confirm all are in WAITING state. This is just a sanity check — rollback is simply setting state back to WAITING.

Step 2: Verify No Jobs Currently Running

SELECT TRIGGER_NAME, TRIGGER_STATE
FROM QRTZ_TRIGGERS
WHERE TRIGGER_NAME LIKE '%BillTransaction%'
  AND TRIGGER_STATE IN ('ACQUIRED', 'EXECUTING', 'BLOCKED');

If any rows return, a job is in progress. Either:

• Wait for it to finish (check again in a few minutes), or
• Proceed — the running job will complete but won’t re-fire

Step 3: Pause the Triggers

UPDATE QRTZ_TRIGGERS
SET TRIGGER_STATE = 'PAUSED'
WHERE TRIGGER_NAME LIKE '%BillTransaction%'
  AND TRIGGER_STATE = 'WAITING';

Step 4: Verify Paused

SELECT TRIGGER_NAME, TRIGGER_GROUP, TRIGGER_STATE
FROM QRTZ_TRIGGERS
WHERE TRIGGER_NAME LIKE '%BillTransaction%';

All matching triggers should show PAUSED.

---

Procedure: Resume BillTransactionJobs

Step 1: Resume the Triggers

UPDATE QRTZ_TRIGGERS
SET TRIGGER_STATE = 'WAITING'
WHERE TRIGGER_NAME LIKE '%BillTransaction%'
  AND TRIGGER_STATE = 'PAUSED';

Step 2: Verify Resumed

SELECT TRIGGER_NAME, TRIGGER_GROUP, TRIGGER_STATE, NEXT_FIRE_TIME
FROM QRTZ_TRIGGERS
WHERE TRIGGER_NAME LIKE '%BillTransaction%';

All triggers should show WAITING with a valid NEXT_FIRE_TIME.

Step 3: Monitor First Execution

After the next scheduled fire time, verify the job ran successfully:

-- Check recent job executions (logs DB)
SELECT TOP 10
    JobName, TriggerName, ExecutionDetails, ExecutionTime, Status
FROM JobExecutionLogs
WHERE JobName LIKE '%BillTransaction%'
ORDER BY ExecutionTime DESC;

Also check the eSHARS admin UI: Jobs tab > search for BillTransaction to confirm jobs are running.

---

Pausing Other Jobs

This same procedure works for any Quartz-managed job. Replace %BillTransaction% with the appropriate trigger name pattern:

Job	Trigger Pattern
BillTransactionJob	%BillTransaction%
EligibilityResponseProcessJob	%Eligibility%Response%
ImportBatchSchedulerJob	%Import%Batch%
CentricityVisitPaymentImportJob	%Centricity%

Code-Registered Jobs

Most jobs other than BillTransactionJob are registered in code at startup (JobService.RegisterJobTypes). If you pause these and the Job Scheduler service restarts, the code may re-register triggers in WAITING state. BillTransactionJobs are safe from this because they are explicitly excluded from code registration.

---

Rollback

If something goes wrong:

• Jobs won’t fire after resume: Check that TRIGGER_STATE is WAITING (not COMPLETE or ERROR). If the trigger’s END_TIME has passed, it may need to be recreated via the eSHARS admin UI.
• Need to stop everything immediately: The Job Scheduler is a Windows service on the prod job scheduler VM. Stop the service: Stop-Service EsharsJobScheduler (nuclear option — stops ALL jobs).

Related

• AthenaPractice Datacenter Migration — first use case for this runbook
• Job Monitoring & Alerts — alert response procedures
• Code reference: Eshars.JobScheduler/JobService.cs, Eshars.JobScheduler/App.config