Skip to content
eSHARS Operations

Incident: 2026-03-24 — SFTP Connection Failure — NAT Gateway IPs Blocked by CrushFTP

Summary

SFTP connections to CrushFTP failed in both production and UAT environments after public IP (PIP) removal as part of the Defender VM remediation effort (HM-8334). Outbound traffic shifted from dedicated PIPs to shared NAT gateway IPs, which CrushFTP’s built-in brute-force protection flagged and blocked due to connection hammering from multiple services behind the same IPs.

FieldValue
SeveritySEV2 — Prod and UAT SFTP connectivity lost
StatusResolved
Reported byRashmi
Investigated byLeon Geldenhuys
Start time2026-03-24 ~04:07 CST
Resolved time2026-03-24 ~06:15 CST
Affected systemSFTP — Eligibility file transfers (CrushFTP)
ImpactEligibility file uploads/downloads blocked for all organizations using SFTP in prod and UAT
Related ticketsHM-8334, DEVOPS-35, DEVOPS-41

Timeline

Time (CST)Event
~04:07Rashmi reports UAT eligibility SFTP failure — SshConnectionException: server response does not contain SSH identification string
~06:00Leon begins investigation; identifies PIP removal (Defender remediation) as trigger
~06:10Root cause confirmed: CrushFTP blocking NAT gateway IPs due to connection hammering detection
~06:15Whitelisted prod and non-prod NAT gateway IPs in CrushFTP; connectivity restored

Root Cause

Primary Cause

The Defender VM remediation work (HM-8334) removed public IPs from several VMs to reduce attack surface. This caused outbound SFTP traffic to route through the VNet’s NAT gateway instead of the VM’s dedicated PIP.

Why CrushFTP Blocked the Traffic

CrushFTP has built-in brute-force / connection-hammering protection. Multiple services behind the same NAT gateway IPs generated a high volume of connections from what appeared to be a single source. CrushFTP interpreted this as hammering and blocked the NAT gateway IPs entirely.

Contributing Factors

  1. PIP removal without SFTP impact assessment — The Defender remediation plan did not account for SFTP endpoints that rely on source IP identity
  2. Shared NAT gateway — Multiple services funnel through the same NAT gateway IPs, amplifying connection counts from a single source IP
  3. CrushFTP hammering protection — No prior documentation of this behavior or its thresholds; the block happened silently
  4. No outbound IP change alerting — No monitoring to detect when a service’s outbound IP changes unexpectedly

Investigation

Symptom

SshConnectionException: server response does not contain SSH identification string

This error indicates the remote server (CrushFTP) is actively refusing the TCP connection before SSH handshake — consistent with an IP-level block.

Diagnosis Steps

  1. Confirmed the affected VMs had their PIPs removed as part of recent Defender work
  2. Verified outbound traffic was now routing through the NAT gateway
  3. Checked CrushFTP admin logs — found NAT gateway IPs in the blocked/banned list
  4. Confirmed the block reason was “connection hammering” threshold exceeded

Resolution

Whitelisted the following NAT gateway IPs in CrushFTP’s allow list so they bypass hammering detection:

  • Prod NAT gateway IPs (2 IPs)
  • Non-prod NAT gateway IPs (2 IPs)

Connection restored immediately after whitelist update.

Lessons Learned

  1. PIP removal has downstream identity effects — When removing PIPs, audit all services that depend on source IP for authentication, whitelisting, or rate-limiting. SFTP servers, firewalls, and partner allow-lists are common dependencies.
  2. CrushFTP’s hammering protection is aggressive — It blocks at the IP level without notification. NAT gateway IPs are high-risk for false positives because they aggregate traffic from many sources.
  3. Document outbound IP dependencies — Maintain a list of services and their expected outbound IPs so changes can be impact-assessed before execution.

Remediation

Immediate (done)

  • Whitelisted prod and non-prod NAT gateway IPs in CrushFTP

Short-term (this sprint)

  • Document all outbound IP dependencies for VMs in the Defender remediation scope
  • Add NAT gateway IP change detection to monitoring (alert if outbound IP shifts)

Long-term

  • Create a pre-flight checklist for PIP removal that includes SFTP/partner IP audit
  • Evaluate CrushFTP hammering thresholds — tune or add NAT-aware exception rules