Changes / Keeper / Vault login errors - resolved
Keeper: Critical incident: Vault login errors - resolved
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Vault login errors - resolved
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2025-06-24 15:27 UTC
Before: not present
After: 2025-06-24 16:01 UTC
Before: not present
After: Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Web Vault (US), Keeper Web Vault (US)
Before: not present
After: We are investigating vault login errors
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: The issue has been resolved. The outage was caused by a high volume of database locks related to a large BreachWatch data update. The engineering team is tracking down the root cause and will be addressing a preventative solution today.
Before: not present
After: At 8:15 AM PST, our DevOps team was alerted to elevated API error rates in production. Upon investigation, we identified the following:
* The production database was experiencing high table lock contention.
* This was triggered by several large-scale BreachWatch data updates.
* A recent backend migration had unintentionally doubled the number of parallel update operations, which exacerbated the issue.
* A database failover was performed to restore service stability.
The issue was fully resolved by 8:40 AM PST. The engineering team has addressed the underlying problem in the update processes and is implementing safeguards to prevent recurrence, including query optimization and improved job orchestration.
We apologize for the disruption.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Vault login errors - resolved (this change) | Major incident |