# Keeper: Critical incident: Vault login errors - resolved

**Summary:** Keeper reported a critical outage causing vault login errors affecting the Keeper Admin Console (US) and Keeper Web Vault (US), starting 2025-06-24 15:27 UTC and resolved at 2025-06-24 16:01 UTC (33 min). Large BreachWatch updates triggered database lock contention, exacerbated by a backend migration doubling parallel updates; engineering planned a preventative solution. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2025-06-24 15:27 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/3f3rt1f6trf0](https://stspg.io/3f3rt1f6trf0)
- [Company: Keeper](https://rewindex.com/company/keeper.md)
- Versions: before (none: first archived) → after incident\_version:311/b8897y888zbt@542d5fa7a61a
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Vault login errors - resolved
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2025-06-24 15:27 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-06-24 16:01 UTC
```

**6. Added: Affected components**

```diff
+ Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Web Vault (US), Keeper Web Vault (US)
```

**7. Added: Update** (investigating) · 2025-06-24 15:27 UTC

```diff
+ We are investigating vault login errors
```

**8. Added: Update** (investigating) · 2025-06-24 15:28 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (resolved) · 2025-06-24 16:01 UTC

```diff
+ The issue has been resolved. The outage was caused by a high volume of database locks related to a large BreachWatch data update. The engineering team is tracking down the root cause and will be addressing a preventative solution today.
```

**10. Added: Update** (postmortem) · 2025-06-24 20:42 UTC

```diff
+ At 8:15 AM PST, our DevOps team was alerted to elevated API error rates in production. Upon investigation, we identified the following: * The production database was experiencing high table lock contention. * This was triggered by several large-scale BreachWatch data updates. * A recent backend migration had unintentionally doubled the number of parallel update operations, which exacerbated the issue. * A database failover was performed to restore service stability. The issue was fully resolved by 8:40 AM PST. The engineering team has addressed the underlying problem in the update processes and is implementing safeguards to prevent recurrence, including query optimization and improved job orchestration. We apologize for the disruption.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-06-24 15:27 UTC | Critical incident: Vault login errors - resolved (this change) | outage |

All Keeper changes: [timeline](https://rewindex.com/company/keeper/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i4a7551e7503405 · Markdown: https://rewindex.com/change/i4a7551e7503405.md · JSON: https://rewindex.com/api/v1/changes/i4a7551e7503405 · Generated 2026-10-18T00:39:29.992Z