Changes / Keeper / Resolved: KeeperPAM Connection Errors in US Data Center
Keeper: Major incident: Resolved: KeeperPAM Connection Errors in US Data Center
Summary
KeeperPAM connection errors in the US Data Center caused a major incident starting 2026-03-23 17:07 UTC, lasting 4 h 37 min and resolving at 2026-03-23 21:45 UTC. Keeper implemented a fix; its postmortem identified a low-level concurrency bug in the Keeper EPM service causing request failures under high simultaneous load and instability in ECS services.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Resolved: KeeperPAM Connection Errors in US Data Center
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-03-23 17:07 UTC
Before: not present
After: 2026-03-23 21:45 UTC
Before: not present
After: Keeper Infrastructure (US Data Center) - Keeper Web Vault (US), Keeper Web Vault (US)
Before: not present
After: We are investigating errors in establishing KeeperPAM connections in the US Data Center.
Before: not present
After: We have identified the cause of the KeeperPAM connection errors which are related to a networking issue within the AWS ECS environment. We are actively troubleshooting with the AWS support team and will update the ticket as soon as there is a status update.
Before: not present
After: A fix has been implemented and we are monitoring the KeeperPAM connection stability. We will update this case with additional details soon.
Before: not present
After: The issue has been resolved. See postmortem page with additional details.
Before: not present
After: At 9:05 AM PST, alerts were triggered for issues affecting KeeperPAM connections managed through Keeper’s ECS deployments in the US-EAST region. There were no recent changes to the application or environment.
Investigation identified a low-level concurrency bug in the Keeper EPM service that caused request failures under high simultaneous load. These failures led to instability in the ECS services supporting KeeperPAM connections.
As a temporary mitigation, we blocked the error condition, restoring KeeperPAM connectivity by approximately 1:00 PM PST.
The engineering team then developed and deployed an updated Keeper Router version to address the underlying issue and prevent EPM agents from triggering server errors. The fix was fully validated by 3:00 PM PST, at which point all KeeperPAM services were stable and operating normally.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Resolved: KeeperPAM Connection Errors in US Data Center (this change) KeeperPAM connection errors in the US Data Center caused a major incident starting 2026-03-23 17:07 UTC, lasting 4 h 37 min and resolving at 2026-03-23 21:45 UTC. Keeper implemented a fix; its postmortem identified a low-level concurrency bug in the Keeper EPM service causing request failures under high simultaneous load and instability in ECS services. | Major incident |