Job Monitoring and Alerting

No-code monitoring solutions for eSHARS background jobs using existing infrastructure.

Overview

eSHARS jobs already emit comprehensive telemetry to multiple destinations. This guide shows how to leverage that data for monitoring and alerting without any code changes.

Current Telemetry Infrastructure

flowchart LR
    subgraph "Job Scheduler"
        Jobs[Quartz.NET Jobs]
        Plugin[JobHistoryLoggerPlugin]
        BaseJob[BaseJob]
    end

    subgraph "Logging Pipeline"
        Log4Net[Log4Net]
        AppLogger[ApplicationLogger]
        Serilog[Serilog]
    end

    subgraph "Destinations"
        AppInsights[(Application Insights)]
        JobExecTable[(JobExecutions Table)]
        JobLogTable[(JobExecutionLogs Table)]
        Console[Console]
    end

    Jobs --> Plugin
    Jobs --> BaseJob
    Plugin -->|"log4net only"| Log4Net
    BaseJob --> AppLogger
    AppLogger --> Log4Net
    AppLogger -->|"message text only"| Serilog
    Log4Net -->|"JobExecutionAppender"| JobExecTable
    Log4Net -->|"JobLogAppender"| JobLogTable
    Serilog --> AppInsights
    Serilog --> Console

Two Logging Pipelines

There are two separate paths — understanding which data lands where is critical for writing correct queries.

Pipeline 1: JobHistoryLoggerPlugin → JobExecutions table

The plugin logs a row when each job starts and completes. It uses raw log4net (NOT ApplicationLogger), so these messages never reach Serilog or App Insights. Properties like jobName, triggerName, ExecutionIdentifier, and JobExecutionStatus are set on log4net’s LogicalThreadContext and written to the DB via the JobExecutionAppender.

Important: Jobs catch their own exceptions internally and do not rethrow them as JobExecutionException. This means the AdditionalDetails column on Status = 2 (Completed) rows is almost always NULL — the plugin only populates it from exception?.ToString(), but the exception never propagates from the job to Quartz.

Pipeline 2: BaseJob / Individual Jobs → ApplicationLogger → JobExecutionLogs table + App Insights

ApplicationLogger wraps log4net and also forwards to Serilog. However, it only forwards the message text and exception — it does NOT push log4net LogicalThreadContext properties to Serilog. This means App Insights receives message, severityLevel, and cloud_RoleName, but NOT customDimensions.jobName, customDimensions.ExecutionIdentifier, etc.

Data Available by Destination

Property	JobExecutions (DB)	JobExecutionLogs (DB)	App Insights
JobName	Yes	—	In message text only
TriggerName	Yes	Yes	—
ExecutionIdentifier	Yes	Yes	—
JobExecutionStatus	Yes (1=Started, 2=Completed)	—	—
OrganizationId	Yes	Yes	—
ProcessId	Yes	Yes	—
Level (INFO/ERROR)	—	Yes	Yes (severityLevel)
Message	ExecutionDetails column	Yes	Yes
Exception	—	Yes	Yes
cloud_RoleName	—	—	Yes (eshars-jobs)

JobExecutionStatus Enum

1 = Started    — Row written when job is about to execute
2 = Completed  — Row written when job finishes (with or without errors)

Note: The C# enum JobExecutionStatus : byte defines Started = 1, Completed = 2. There is no 0 value.

---

Application Insights Queries

Access via: Azure Portal > Application Insights > Logs

Limitation: Log4net context properties (jobName, triggerName, ExecutionIdentifier, OrganizationId) are NOT available in App Insights. Job names must be extracted from message text. For queries that need these properties, use the Database Queries section below.

Job Failures (Last 24 Hours)

traces
| where timestamp > ago(24h)
| where cloud_RoleName == "eshars-jobs"
| where severityLevel >= 3  // Error or higher
| extend jobName = extract(@"^(\w+Job)\s", 1, message)
| project timestamp, jobName, message
| order by timestamp desc

Job Execution Summary (Last 7 Days)

Uses BaseJob’s log messages: "{jobName} started" and "{jobName} completed in {seconds} seconds".

traces
| where timestamp > ago(7d)
| where cloud_RoleName == "eshars-jobs"
| where message matches regex @"^\w+Job (started|completed)"
    or severityLevel >= 3
| extend jobName = extract(@"^(\w+Job)", 1, message)
| where isnotempty(jobName)
| summarize
    TotalRuns = countif(message endswith "started"),
    Completed = countif(message contains "completed in"),
    Errors = countif(severityLevel >= 3)
    by jobName
| extend SuccessRate = round(100.0 * Completed / TotalRuns, 1)
| order by Errors desc

Long-Running Jobs

BaseJob logs completion with duration: "{jobName} completed in {seconds} seconds".

traces
| where timestamp > ago(24h)
| where cloud_RoleName == "eshars-jobs"
| where message matches regex @"completed in [\d.]+ seconds"
| extend jobName = extract(@"^(\w+Job)", 1, message)
| extend durationSeconds = todouble(extract(@"completed in ([\d.]+) seconds", 1, message))
| extend durationMinutes = round(durationSeconds / 60.0, 1)
| where durationMinutes > 30  // Adjust threshold as needed
| project timestamp, jobName, durationMinutes, message
| order by durationMinutes desc

Stuck Jobs (Started But Never Completed)

Note: ExecutionIdentifier is not available in App Insights, so precise per-execution matching is not possible. For accurate stuck job detection, use the QRTZ_FIRED_TRIGGERS database query below. This query provides an approximation by matching job names.

let starts = traces
| where timestamp > ago(4h)
| where timestamp < ago(1h)  // Started more than 1 hour ago
| where cloud_RoleName == "eshars-jobs"
| where message endswith "started"
| extend jobName = extract(@"^(\w+Job)", 1, message)
| where isnotempty(jobName)
| summarize startTime = max(timestamp) by jobName;

let completions = traces
| where timestamp > ago(4h)
| where cloud_RoleName == "eshars-jobs"
| where message contains "completed in"
| extend jobName = extract(@"^(\w+Job)", 1, message)
| where isnotempty(jobName)
| summarize lastCompleted = max(timestamp) by jobName;

starts
| join kind=leftanti completions on jobName
| extend runningMinutes = datetime_diff('minute', now(), startTime)
| project jobName, startTime, runningMinutes
| order by runningMinutes desc

Exception Details

traces
| where timestamp > ago(24h)
| where cloud_RoleName == "eshars-jobs"
| where severityLevel >= 3
| extend jobName = extract(@"^(\w+Job)", 1, message)
| project timestamp, jobName, message
| order by timestamp desc

---

Database Queries

Connect to: ESHARSLOGS database

Recent Job Failures

Jobs catch their own exceptions and log them to JobExecutionLogs via Log.Error(). Query this table for actual error details.

SELECT TOP 50
    jel.Date,
    jel.TriggerName,
    jel.Message,
    jel.Exception,
    jel.OrganizationId,
    jel.ExecutionIdentifier
FROM JobExecutionLogs jel
WHERE jel.Level = 'ERROR'
  AND jel.Date > DATEADD(DAY, -7, GETDATE())
ORDER BY jel.Date DESC;

Job Execution History with Duration

WITH JobRuns AS (
    SELECT
        ExecutionIdentifier,
        JobName,
        TriggerName,
        OrganizationId,
        MIN(CASE WHEN Status = 1 THEN ExecutionTime END) AS StartTime,  -- 1 = Started
        MAX(CASE WHEN Status = 2 THEN ExecutionTime END) AS EndTime     -- 2 = Completed
    FROM JobExecutions
    WHERE ExecutionTime > DATEADD(DAY, -7, GETDATE())
      AND ExecutionIdentifier IS NOT NULL
    GROUP BY ExecutionIdentifier, JobName, TriggerName, OrganizationId
),
ErrorCounts AS (
    SELECT ExecutionIdentifier, COUNT(*) AS ErrorCount
    FROM JobExecutionLogs
    WHERE Level = 'ERROR'
      AND Date > DATEADD(DAY, -7, GETDATE())
    GROUP BY ExecutionIdentifier
)
SELECT
    jr.JobName,
    jr.TriggerName,
    jr.OrganizationId,
    jr.StartTime,
    jr.EndTime,
    DATEDIFF(MINUTE, jr.StartTime, jr.EndTime) AS DurationMinutes,
    CASE
        WHEN jr.EndTime IS NULL THEN 'Incomplete'
        WHEN ec.ErrorCount > 0 THEN CONCAT('Completed (', ec.ErrorCount, ' errors)')
        ELSE 'Success'
    END AS Result
FROM JobRuns jr
LEFT JOIN ErrorCounts ec ON jr.ExecutionIdentifier = ec.ExecutionIdentifier
WHERE jr.StartTime IS NOT NULL
ORDER BY jr.StartTime DESC;

Jobs Currently Running (Stuck Job Detection)

-- Quartz.NET fired triggers table (lives in ESHARS database, not ESHARSLOGS)
SELECT
    ej.JOB_NAME,
    ej.JOB_GROUP,
    ej.TRIGGER_NAME,
    ej.STATE,
    DATEADD(MILLISECOND, ej.FIRED_TIME % 1000,
        DATEADD(SECOND, ej.FIRED_TIME / 1000, '1970-01-01')) AS FiredTimeUtc,
    DATEDIFF(MINUTE,
        DATEADD(MILLISECOND, ej.FIRED_TIME % 1000,
            DATEADD(SECOND, ej.FIRED_TIME / 1000, '1970-01-01')),
        GETUTCDATE()) AS RunningMinutes
FROM QRTZ_FIRED_TRIGGERS ej
WHERE ej.STATE = 'EXECUTING'
ORDER BY ej.FIRED_TIME ASC;

Job Health Report (Last 7 Days)

Note: Because jobs catch their own exceptions and always reach Status = 2 (Completed), a simple “has ERROR logs = failed” check produces misleading results (e.g., 100% failure rates for jobs that process thousands of records but log a handful of errors). This query distinguishes between truly incomplete runs and completed runs that logged errors, and shows error volume per job.

WITH CompletedRuns AS (
    SELECT DISTINCT ExecutionIdentifier, JobName
    FROM JobExecutions
    WHERE ExecutionTime > DATEADD(DAY, -7, GETDATE())
      AND ExecutionIdentifier IS NOT NULL
      AND Status = 2  -- Completed
),
StartedRuns AS (
    SELECT DISTINCT ExecutionIdentifier, JobName
    FROM JobExecutions
    WHERE ExecutionTime > DATEADD(DAY, -7, GETDATE())
      AND ExecutionIdentifier IS NOT NULL
      AND Status = 1  -- Started
),
ErrorCounts AS (
    SELECT
        ExecutionIdentifier,
        COUNT(*) AS ErrorCount
    FROM JobExecutionLogs
    WHERE Level = 'ERROR'
      AND Date > DATEADD(DAY, -7, GETDATE())
      AND ExecutionIdentifier IS NOT NULL
    GROUP BY ExecutionIdentifier
),
JobStats AS (
    SELECT
        sr.JobName,
        COUNT(DISTINCT sr.ExecutionIdentifier) AS TotalRuns,
        COUNT(DISTINCT cr.ExecutionIdentifier) AS CompletedRuns,
        COUNT(DISTINCT CASE WHEN cr.ExecutionIdentifier IS NULL
              THEN sr.ExecutionIdentifier END) AS IncompleteRuns,
        COUNT(DISTINCT CASE WHEN ec.ExecutionIdentifier IS NOT NULL
              THEN sr.ExecutionIdentifier END) AS RunsWithErrors,
        SUM(ISNULL(ec.ErrorCount, 0)) AS TotalErrors
    FROM StartedRuns sr
    LEFT JOIN CompletedRuns cr ON sr.ExecutionIdentifier = cr.ExecutionIdentifier
    LEFT JOIN ErrorCounts ec ON sr.ExecutionIdentifier = ec.ExecutionIdentifier
    GROUP BY sr.JobName
)
SELECT
    JobName,
    TotalRuns,
    CompletedRuns,
    IncompleteRuns,
    RunsWithErrors,
    TotalErrors,
    CAST(100.0 * RunsWithErrors / NULLIF(TotalRuns, 0) AS DECIMAL(5,2)) AS PctRunsWithErrors,
    CAST(1.0 * TotalErrors / NULLIF(TotalRuns, 0) AS DECIMAL(10,2)) AS AvgErrorsPerRun
FROM JobStats
ORDER BY IncompleteRuns DESC, AvgErrorsPerRun DESC;

Reading the results:

Column	Meaning
CompletedRuns	Jobs that finished normally
IncompleteRuns	Jobs that started but never completed (true failures — likely 0 due to internal exception handling)
RunsWithErrors	Completed runs that logged at least one error
TotalErrors	Total error log entries across all runs
AvgErrorsPerRun	Error volume — distinguishes “1 bad record” from “systematic issue”

Detailed Logs for a Specific Execution

-- Replace 'your-execution-guid' with actual ExecutionIdentifier
SELECT
    jel.Date,
    jel.Level,
    jel.Logger,
    jel.Message,
    jel.Exception,
    jel.TriggerName,
    jel.OrganizationId
FROM JobExecutionLogs jel
WHERE jel.ExecutionIdentifier = 'your-execution-guid'
ORDER BY jel.Date ASC;

Jobs Not Run Recently (Possible Scheduling Issues)

WITH LastRuns AS (
    SELECT
        JobName,
        MAX(ExecutionTime) AS LastRunTime
    FROM JobExecutions
    WHERE Status = 1  -- Started
    GROUP BY JobName
)
SELECT
    JobName,
    LastRunTime,
    DATEDIFF(HOUR, LastRunTime, GETDATE()) AS HoursSinceLastRun
FROM LastRuns
WHERE DATEDIFF(HOUR, LastRunTime, GETDATE()) > 24
ORDER BY HoursSinceLastRun DESC;

---

Setting Up Alerts

Application Insights Alert Rules

1. Navigate to Azure Portal > Application Insights > Alerts > New Alert Rule

2. Condition: Custom log search

3. Use these alert queries:

Alert: Job Failure

traces
| where cloud_RoleName == "eshars-jobs"
| where severityLevel >= 3
| extend jobName = extract(@"^(\w+Job)", 1, message)
| project timestamp, jobName, message

• Threshold: Greater than 0
• Frequency: Every 5 minutes
• Period: Last 15 minutes

Alert: Stuck Job

Note: For reliable stuck job detection, use the QRTZ_FIRED_TRIGGERS SQL query via a SQL Agent job instead. App Insights does not have ExecutionIdentifier for precise matching.

let starts = traces
| where timestamp > ago(2h)
| where timestamp < ago(30m)
| where cloud_RoleName == "eshars-jobs"
| where message endswith "started"
| extend jobName = extract(@"^(\w+Job)", 1, message)
| where isnotempty(jobName)
| summarize startTime = max(timestamp) by jobName;

let completions = traces
| where timestamp > ago(2h)
| where cloud_RoleName == "eshars-jobs"
| where message contains "completed in"
| extend jobName = extract(@"^(\w+Job)", 1, message)
| where isnotempty(jobName)
| summarize lastCompleted = max(timestamp) by jobName;

starts
| join kind=leftanti completions on jobName
| count

• Threshold: Greater than 0
• Frequency: Every 15 minutes
• Period: Last 2 hours

Alert: High Failure Rate

let allJobs = traces
| where timestamp > ago(1h)
| where cloud_RoleName == "eshars-jobs"
| where message endswith "started"
| summarize Total = count();

let errors = traces
| where timestamp > ago(1h)
| where cloud_RoleName == "eshars-jobs"
| where severityLevel >= 3
| summarize Failed = count();

allJobs
| extend dummy = 1
| join kind=inner (errors | extend dummy = 1) on dummy
| extend FailureRate = 100.0 * Failed / Total
| where FailureRate > 20  // Alert if >20% failure rate

• Threshold: Greater than 0
• Frequency: Every 30 minutes
• Period: Last 1 hour

SQL Server Agent Alerts

Create a SQL Agent job that runs every 15 minutes:

-- Check for recent job failures and send email alert
DECLARE @FailureCount INT;
DECLARE @FailureDetails NVARCHAR(MAX);

SELECT @FailureCount = COUNT(DISTINCT ExecutionIdentifier)
FROM JobExecutionLogs
WHERE Level = 'ERROR'
  AND Date > DATEADD(MINUTE, -15, GETDATE());

IF @FailureCount > 0
BEGIN
    SELECT @FailureDetails = STRING_AGG(
        CONCAT(TriggerName, ' at ', FORMAT(Date, 'yyyy-MM-dd HH:mm:ss'), ': ', LEFT(Message, 200)),
        CHAR(13) + CHAR(10)
    )
    FROM (
        SELECT DISTINCT
            TriggerName,
            MIN(Date) AS Date,
            MIN(Message) AS Message
        FROM JobExecutionLogs
        WHERE Level = 'ERROR'
          AND Date > DATEADD(MINUTE, -15, GETDATE())
        GROUP BY TriggerName, ExecutionIdentifier
    ) failures;

    EXEC msdb.dbo.sp_send_dbmail
        @profile_name = 'YourMailProfile',
        @recipients = 'team@example.com',
        @subject = 'eSHARS Job Failure Alert',
        @body = @FailureDetails;
END

---

Power BI Dashboard

Connect Power BI to the ESHARSLOGS database for visual monitoring.

Recommended Visualizations

1. Job Health Summary Card

   • Total jobs run today
   • Success rate percentage
   • Active failures

2. Failure Trend Line Chart

   • X-axis: Date/Time
   • Y-axis: Failure count
   • Group by: Job Name

3. Job Duration Box Plot

   • Identify outliers and performance degradation

4. Heatmap: Failures by Hour/Day

   • Identify problematic time windows

Sample Power BI Query

WITH Runs AS (
    SELECT
        ExecutionIdentifier,
        JobName,
        MIN(CASE WHEN Status = 1 THEN ExecutionTime END) AS StartTime,
        MAX(CASE WHEN Status = 2 THEN ExecutionTime END) AS EndTime
    FROM JobExecutions
    WHERE ExecutionTime > DATEADD(DAY, -30, GETDATE())
      AND ExecutionIdentifier IS NOT NULL
    GROUP BY ExecutionIdentifier, JobName
),
ErrorCounts AS (
    SELECT ExecutionIdentifier, COUNT(*) AS ErrorCount
    FROM JobExecutionLogs
    WHERE Level = 'ERROR'
      AND Date > DATEADD(DAY, -30, GETDATE())
    GROUP BY ExecutionIdentifier
)
SELECT
    CAST(r.StartTime AS DATE) AS ExecutionDate,
    DATEPART(HOUR, r.StartTime) AS ExecutionHour,
    r.JobName,
    COUNT(DISTINCT r.ExecutionIdentifier) AS TotalRuns,
    COUNT(DISTINCT CASE WHEN r.EndTime IS NULL THEN r.ExecutionIdentifier END) AS IncompleteRuns,
    COUNT(DISTINCT CASE WHEN ec.ExecutionIdentifier IS NOT NULL THEN r.ExecutionIdentifier END) AS RunsWithErrors,
    SUM(ISNULL(ec.ErrorCount, 0)) AS TotalErrors
FROM Runs r
LEFT JOIN ErrorCounts ec ON r.ExecutionIdentifier = ec.ExecutionIdentifier
WHERE r.StartTime IS NOT NULL
GROUP BY CAST(r.StartTime AS DATE), DATEPART(HOUR, r.StartTime), r.JobName
ORDER BY ExecutionDate DESC, ExecutionHour;

---

Application Insights Environments

Each environment has a dedicated Application Insights account:

Environment	Application Insights Account
Development	esharsweb-dev
QA	esharsweb-qa
Regression	esharsweb-regression
UAT	esharsweb-uat
Training	esharsweb-training
Production	eshars-prod

All accounts are in the rg-eshars-nonprod resource group (non-prod) or rg-eshars-prod (production).

Log Segregation

Web and job logs are in the same Application Insights environment but segregated by cloud_RoleName:

• eshars-web — eSHARS web application traces
• eshars-jobs — Job Scheduler traces

Logging Configuration

Both the web application and Job Scheduler use Log4Net and Serilog. Logging is controlled by web.config / app.config settings:

<add key="EnableConsoleLogging" value="true" />
<add key="EnableAppInsightsLogging" value="true" />

For detailed logging architecture, see Logging.

Existing Monitoring

eSHARS already has some monitoring in place:

SystemHealthNotificationJob

Runs every 30 minutes and sends Teams alerts for:

• Delayed import batches
• Delayed eligibility requests

Configuration keys:

• SendTeamsNotificationFlag - Global enable/disable
• SystemHealthNotificationFlagImports - Import monitoring
• SystemHealthNotificationWebhookURLImports - Teams webhook URL
• SystemHealthNotificationImportsTimeInHoursBeforeSendingNotification - Delay threshold

BillTransactionJob Notifications

Sends Teams notifications for billing job status.

StudentServiceActivationNotificationJob

Sends notifications for new student service activations.

---

Recommended Monitoring Strategy

Priority	Monitor	Method	Alert Channel
P1	Job failures	App Insights Alert	Teams/PagerDuty
P1	Stuck jobs (>30 min)	SQL Agent + QRTZ_FIRED_TRIGGERS	Teams/PagerDuty
P2	High failure rate (>20%)	App Insights Alert	Teams/Email
P2	Jobs not running on schedule	SQL Agent Job	Email
P3	Performance degradation	Power BI Dashboard	Weekly review
P3	Trend analysis	Power BI Dashboard	Weekly review

---

Troubleshooting

Finding logs for a failed job

1. Get the ExecutionIdentifier from the JobExecutionLogs or JobExecutions table
2. Query detailed logs:

Database (primary — has full structured data):

SELECT * FROM JobExecutionLogs
WHERE ExecutionIdentifier = 'your-guid-here'
ORDER BY Date ASC;

App Insights (message text only — no ExecutionIdentifier filter available):

traces
| where timestamp > ago(24h)
| where cloud_RoleName == "eshars-jobs"
| where severityLevel >= 3
| where message contains "YourJobName"
| order by timestamp asc

Common failure patterns

Symptom	Likely Cause	Investigation
SFTP connection errors	Network/credentials	Check SFTP settings
Database timeouts	Long-running queries	Check query plans
Memory exceptions	Large batch sizes	Review batch configuration
Concurrent execution errors	Lock contention	Check QRTZ_FIRED_TRIGGERS table

---

Known Issues and Recommended Improvements

Exception Handling Inconsistencies

BaseJob.Execute() has a top-level try/catch that rethrows exceptions, which would allow the JobHistoryLoggerPlugin to capture them in the AdditionalDetails column of JobExecutions. However, every job implementation catches exceptions internally in ExecuteJob() and does not rethrow. This silently breaks the plugin’s error-capture mechanism.

Job	Pattern	Rethrows?
ProcessImportBatchJob	Task.Run(() => { try { ... } catch { Log.Error(...) } }).Wait() — exception caught inside Task.Run, returns Task.CompletedTask regardless	No
BillTransactionJob	Multiple nested try/catch blocks, uses a jobSuccessful boolean flag internally but never throws	No
EligibilityResponseProcessJob	Task.Run(() => { try { ... } catch { Log.Error(...) } }).Wait() — continues processing remaining files after per-file failures	No
ImportBatchSchedulerJob	Simple try/catch around entire ExecuteJob, logs error but swallows it	No

Impact:

• JobExecutions.AdditionalDetails on Completed rows is always NULL, making the JobExecutions table unreliable for failure detection
• Quartz.NET sees every job as successful, so any Quartz-level retry or failure-handling configuration has no effect
• Failure detection requires querying JobExecutionLogs for Level = 'ERROR' entries, which is indirect

ApplicationLogger Does Not Forward Context Properties to Serilog

ApplicationLogger wraps a log4net ILog and also calls Serilog’s static Log methods. But it only passes the message string and exception — it does not push any of the LogicalThreadContext properties (jobName, triggerName, ExecutionIdentifier, OrganizationId, etc.) into Serilog’s LogContext.

Impact:

• App Insights traces contain only message, severityLevel, and cloud_RoleName
• Job names must be extracted from message text using regex, which is fragile
• Cannot filter or group by ExecutionIdentifier, OrganizationId, or TriggerName in App Insights
• Cannot correlate start/completion events for the same execution in App Insights

Task.Run Anti-Pattern

ProcessImportBatchJob and EligibilityResponseProcessJob wrap their logic in Task.Run(() => { ... }).Wait() with an internal try/catch. Even if they did rethrow inside Task.Run, the exception would be wrapped in an AggregateException by .Wait(). Combined with the internal catch, this pattern ensures exceptions never propagate.

Recommended Improvements

1. Propagate exceptions from jobs (High Impact)

Remove or rethrow in the internal try/catch blocks so BaseJob and the plugin can see failures. This would make JobExecutions.AdditionalDetails work as intended.

// Current (broken)
protected override Task ExecuteJob(IJobExecutionContext context)
{
    try { /* ... */ }
    catch (Exception e)
    {
        Log.Error($"Job failed: {e.Message}", e);
        // exception swallowed — plugin never sees it
    }
}

// Recommended
protected override Task ExecuteJob(IJobExecutionContext context)
{
    try { /* ... */ }
    catch (Exception e)
    {
        Log.Error($"Job failed: {e.Message}", e);
        throw;  // let BaseJob and plugin handle it
    }
}

For jobs that process multiple items (e.g., EligibilityResponseProcessJob processing multiple files), catch per-item exceptions to continue processing, but throw an aggregate exception at the end if any items failed.

2. Forward log4net properties to Serilog (Medium Impact)

Enrich Serilog’s LogContext with the relevant properties in ApplicationLogger.LogMessage() so they appear as customDimensions in App Insights.

private void LogMessage(log4net.Core.Level level, object message, Exception exception)
{
    _log4NetLogger.Logger.Log(typeof(ApplicationLogger), level, message, exception);

    string messageText = message?.ToString() ?? "null";

    // Push log4net properties into Serilog's LogContext
    using (LogContext.PushProperty("jobName", LogicalThreadContext.Properties["jobName"]))
    using (LogContext.PushProperty("triggerName", LogicalThreadContext.Properties["triggerName"]))
    using (LogContext.PushProperty("ExecutionIdentifier", LogicalThreadContext.Properties["ExecutionIdentifier"]))
    using (LogContext.PushProperty("OrganizationId", LogicalThreadContext.Properties["OrganizationId"]))
    {
        // existing switch statement...
    }
}

This would make all the customDimensions-based App Insights queries work.

3. Remove Task.Run wrapping (Low Impact)

ProcessImportBatchJob and EligibilityResponseProcessJob should use async/await directly instead of Task.Run(() => { ... }).Wait(). The ExecuteJob method is already async Task, so wrapping synchronous code in Task.Run adds unnecessary complexity and obscures exception propagation.

---

Related

• Jobs Architecture
• Database Schema
• Incident Response
• Troubleshooting - Import Failures