Companies / Keeper / incidents
Keeper: Login errors in US region
Keeper reported a major US-region login outage affecting Keeper Admin Console (US) and Keeper Web Vault (US), starting 2025-02-10 17:30 UTC and resolved at 2025-02-10 17:49 UTC (19 min). The postmortem attributed failed new login sessions to a database lock in the primary writer instance and said the issue resolved without intervention; a backend update was planned.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Keeper Admin Console (US)Keeper Infrastructure (US Data Center) - Keeper Admin Console (US)Keeper Infrastructure (US Data Center) - Keeper Web Vault (US)Keeper Web Vault (US)
Timeline of updates
- identified
We are aware of a high number of errors in the US region. The issue will be fixed shortly.
- resolved
We have resolved the issue and plan to deploy a backend update shortly.
- postmortem
At 9:21AM PST we received notifications of API errors coming from our backend systems. The issue resolved itself without any required action at 9:31AM PST. During that 10 minute period, there was a database lock in the primary writer instance causing new login sessions to fail. Recently, Keeper upgraded all production RDS Aurora databases to MySQL 8.x. This updated database engine is sensitive to one particular SQL query that was causing large table scans to occur. The code changes and QA process was already under way when this issue occurred. The backend code change which addresses this issue passed QA at 12:48PM PST and we released the fix to prevent this issue from occurring again \(Release Jira ticket REL-4633\). We’re keeping a close eye and will issue additional updates as necessary.
Rewindex changes for this incident
- Major incident: Login errors in US region 2025-02-10 17:30 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Keeper profile · Recent Keeper incidents · Markdown · JSON