Incident: 2026-02-20 — Job Scheduler OutOfMemoryException

Incident: 2026-02-20 — Job Scheduler OutOfMemoryException

Summary

EsharsJobScheduler.exe on vm-jobscheduler crashed with System.OutOfMemoryException after exhausting 32GB of RAM. Investigation revealed three layers of systemic memory management issues affecting all jobs, with the billing transaction job as the likely trigger.

Field	Value
Severity	SEV2 — Service crash
Status	Root cause identified, fixes pending
Reported by	Windows Event Log (Application, Event 1026)
Investigated by	Leon Geldenhuys
Occurred	2026-02-20 18:27:45 UTC
Resolved	Pending — see HM-8478
Affected system	Job Scheduler (Quartz.NET Windows Service)
Impact	All scheduled jobs stopped — billing, eligibility, imports, health monitoring

---

Event Log Entry

Log Name:    Application
Source:      .NET Runtime
Event ID:    1026
Level:       Error
Computer:    vm-jobscheduler

Application: EsharsJobScheduler.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at System.Threading.ExecutionContext.CreateCopy()
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.TimerQueue.AppDomainTimerCallback(Int32)

Stack trace is misleading

The OOM occurred in TimerQueueTimer.CallCallback — this is the Quartz scheduler’s timer trying to fire the next job. Memory was already exhausted; the timer thread was just the one that failed. The actual memory consumer is one or more of the batch-processing jobs.

---

Root Cause Analysis

Three compounding layers of memory issues were identified:

Layer 1: DI Scope Disposal Bug (affects ALL jobs)

File: Eshars.JobScheduler/Infrastructure/LifestyleScopeJobDecorator.cs

LifestyleScopeJobDecorator.cs — the bug

Task IJob.Execute(IJobExecutionContext context)
{
    var t = Task.Run(() =>
    {
        using (ThreadScopedLifestyle.BeginScope(_container))
        {
            var job = _decorateeFactory();
            job.Execute(context);  // NOT awaited — scope disposes immediately
        }
    });
    return t;
}

BaseJob.Execute() is async Task, but the decorator calls it without await. The using block exits after the synchronous portion returns, disposing the DI scope while the async job continues running. Every DbContext resolved after the first await in any job is created outside the disposed scope and may never be properly disposed.

This is the root cause

Every job in the scheduler is affected. DbContexts leak across the lifetime of the service process, accumulating tracked entities indefinitely.

Fix: await job.Execute(context) — see HM-8479

---

Layer 2: AuditDbContext ChangeTracker Accumulation

File: Eshars.Core.DataModel/Abstract/AuditDbContext.cs

Every SaveChanges() call:

1. Iterates all ChangeTracker.Entries() — including previously-saved entities
2. Calls Auditor.Audit(entity) for each, creating audit log records
3. Saves main context, then saves audit context
4. Never clears tracked entities — they accumulate indefinitely

Combined with EF6 settings:

Configuration.LazyLoadingEnabled = true;   // creates proxy wrappers
Configuration.ProxyCreationEnabled = true;  // proxies pin entities in memory

In a web request (short-lived), this is harmless. In a job processing thousands of records over minutes/hours, entities accumulate without bound.

Fix: Detach entities after save — see HM-8480

---

Layer 3: BillTransactionJob — Unbounded Data Loading

File: Eshars.JobScheduler/JobDefinitions/BillTransactionJob.cs

The billing job has several memory-intensive patterns:

Unfiltered Visit Query

Line 332 — loads ALL visits

var clinicianAllVisitQueryable = _ediUnitOfWork.Repository<Visit>().Get()  // ALL visits
    .Join(visitClinicianList, ...);
var clinicianAllVisitIds = clinicianAllVisitQueryable.Select(v => v.Id).ToList();  // materialize

.Get() returns the entire Visit table unfiltered. With hundreds of thousands of visits, this materializes a massive list into memory.

Massive String Allocation

Line 341 — joins all IDs into one string

visitIdsAvailableForProcessing = _visitService.GetOverlappingVisitList(
    string.Join(", ", clinicianAllVisitIds)  // 500K IDs = ~5MB string
).Select(v => v.VisitId).ToList();

All visit IDs are concatenated into a single comma-separated string and passed to a stored procedure. With 500K+ visits, this is a 5MB+ string allocation.

Parallel Processing

Lines 379-393 — concurrent DbContexts

var studentVisitTasks = new Task[studentBatch.Count()];
for (var i = 0; i < studentBatch.Count(); i++)
{
    studentVisitTasks[i] = Task.Run(async () =>
    {
        var status = await ProcessVisitsForStudent(studentId, execution.Id);
        totalVisitsBilled += status.BilledVisitsCount;  // race condition!
    });
}
Task.WaitAll(studentVisitTasks);

Each parallel task creates its own ediUnitOfWork with full DbContext. With StudentProcessBatchSize = 10, that’s 10 concurrent DbContexts each materializing visits via .ToList().

Thread safety bug

totalVisitsBilled += is not thread-safe. Multiple threads incrementing the same variable without Interlocked.Add() can cause data corruption.

Fix: Filter queries, batch IDs, fix thread safety — see HM-8481

---

Contributing: SystemHealthNotificationJob

File: Eshars.JobScheduler/JobDefinitions/SystemHealthNotificationJob.cs

Runs every 30 minutes. Executes 16 separate .ToList() calls in a single loop, loading entire staging tables into memory just to count records:

// This pattern repeats 16 times for each import type:
var stagedList = _importService.GetStagingStudentDetails(id).ToList();
completedRecordCount = stagedList.Where(x => x.Status == "Completed").Count();
stagedRecordCount = stagedList.Where(x => x.Status == "Staged").Count();

A SELECT COUNT(*) WHERE Status = 'Completed' would use zero application memory.

Fix: Use IQueryable.Count() — see HM-8482

---

Memory Growth Model

graph TD
    A[Service starts] --> B[Jobs run on schedule]
    B --> C[LifestyleScopeJobDecorator<br/>disposes scope early]
    C --> D[DbContexts leak<br/>not disposed properly]
    D --> E[ChangeTracker accumulates<br/>entities across saves]
    E --> F[EF6 proxies pin<br/>entities in memory]
    F --> G[BillTransactionJob loads<br/>500K+ visits into memory]
    G --> H[String.Join creates<br/>5MB+ strings]
    H --> I[Parallel tasks create<br/>10 concurrent DbContexts]
    I --> J[GC cannot collect<br/>pinned objects]
    J --> K[32GB exhausted]
    K --> L[OOM on next<br/>timer callback]

    style C fill:#ff6b6b,color:#fff
    style D fill:#ff6b6b,color:#fff
    style E fill:#ffa94d,color:#fff
    style G fill:#ffa94d,color:#fff

---

Diagnostic Queries

Use these queries against HisdDbLog to investigate future occurrences:

Jobs running before crash

SELECT TOP 50
    Date, TriggerName, Level, Logger,
    LEFT(Message, 200) AS MessagePreview,
    LEFT(Exception, 300) AS ExceptionPreview
FROM JobExecutionLog
WHERE Date BETWEEN DATEADD(HOUR, -1, @crash_time) AND @crash_time
ORDER BY Date DESC;

Jobs that started but never completed (killed by OOM)

SELECT jel.Date AS StartTime, jel.TriggerName, jel.Message
FROM JobExecutionLog jel
WHERE jel.Date BETWEEN DATEADD(HOUR, -1, @crash_time) AND @crash_time
  AND jel.Message LIKE '%started%'
  AND jel.TriggerName NOT IN (
    SELECT TriggerName FROM JobExecutionLog
    WHERE Date BETWEEN DATEADD(HOUR, -1, @crash_time) AND DATEADD(MINUTE, 5, @crash_time)
      AND Message LIKE '%completed%'
  )
ORDER BY jel.Date;

---

Fix Priority

Priority	Ticket	Fix	Risk
P0	HM-8479	Await async job in LifestyleScopeJobDecorator	High — affects all jobs
P1	HM-8480	Clear ChangeTracker after SaveChanges	Medium — callers may depend on tracking
P1	HM-8481	BillTransactionJob: filter visits, batch IDs	High — billing pipeline
P2	HM-8482	SystemHealthNotificationJob: IQueryable Count	Low — monitoring only
P2	HM-8483	Disable LazyLoading in job scheduler	Medium — needs eager load audit

Parent epic: HM-8477 — eSHARS Application Performance & Reliability

---

Lessons Learned

1. Long-running services need different EF patterns than web apps. LazyLoading, ProxyCreation, and ChangeTracker accumulation are acceptable for short-lived web requests but cause unbounded growth in batch services.

2. async without await is a silent killer. The scope disposal bug has likely been present since the scheduler was written, slowly leaking memory. The crash only happened when data volume grew large enough.

3. Memory pressure is gradual, crash is sudden. The OOM didn’t happen in the code that consumed memory — it happened in an unrelated timer thread. Always look at the broader system, not just the crash stack trace.

4. .ToList() is the most dangerous LINQ method in batch code. Every call materializes entire result sets. Prefer IQueryable operations that translate to SQL, and only materialize when you need to iterate.