# Keeper: Critical incident: Keeper API website load time and API errors in US region

**Summary:** Keeper reported a critical incident involving slow responses and HTTP 504 errors, listing Admin Console and Web Vault components in EU, GovCloud and US, plus the US website. It lasted from 2024-09-24 13:43 UTC to 2024-09-24 14:43 UTC (1 h 0 min); NGINX network traffic overload caused errors, and a resolution was rolled out. _(machine-written, sol)_
- Kind: Major incident · Label: degradation (model (Jev))
- Changed: 2024-09-24 13:43 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/syypkzh0jbfn](https://stspg.io/syypkzh0jbfn)
- [Company: Keeper](https://rewindex.com/company/keeper.md)
- Versions: before (none: first archived) → after incident\_version:311/9gq37x2zw8q7@918ca9c89332
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Keeper API website load time and API errors in US region
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2024-09-24 13:43 UTC
```

**5. Added: Resolved time**

```diff
+ 2024-09-24 14:43 UTC
```

**6. Added: Affected components**

```diff
+ Keeper Admin Console (EU), Keeper Admin Console (GovCloud), Keeper Admin Console (US), Keeper Infrastructure (EU Data Center) - Keeper Admin Console (EU), Keeper Infrastructure (EU Data Center) - Keeper Web Vault (EU), Keeper Infrastructure (GovCloud) - Keeper Admin Console (GovCloud), Keeper Infrastructure (GovCloud) - Keeper Web Vault (GovCloud), Keeper Infrastructure (US Data Center) - Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Security Website (US), Keeper Infrastructure (US Data Center) - Keeper Web Vault (US), Keeper Security Website (US), Keeper Web Vault (EU), Keeper Web Vault (GovCloud), Keeper Web Vault (US)
```

**7. Added: Update** (investigating) · 2024-09-24 13:43 UTC

```diff
+ We are currently investigating reports of slow server response times and request timeouts
```

**8. Added: Update** (identified) · 2024-09-24 14:07 UTC

```diff
+ Root cause has been identified and we are actively working on a resolution.
```

**9. Added: Update** (monitoring) · 2024-09-24 14:28 UTC

```diff
+ A resolution has been rolled out - the team is monitoring system performance.
```

**10. Added: Update** (monitoring) · 2024-09-24 14:34 UTC

```diff
+ We are continuing to monitor for any further issues.
```

**11. Added: Update** (resolved) · 2024-09-24 14:43 UTC

```diff
+ This incident has been resolved.
```

**12. Added: Update** (postmortem) · 2024-09-24 23:17 UTC

```diff
+ On Tues, Sept 24 at 6:06AM PDT, the DevOps Engineering team started receiving notifications regarding API slowness and HTTP 504 errors coming from the Keeper AWS infrastructure. After an investigation, it was determined that our NGINX instances were throwing the errors due to an overload of network traffic. It was then determined that the network traffic increase was caused by an update to the Keeper Desktop application which was published on Monday, the day before. The DevOps team scaled the AWS infrastructure several times over the period of 60 minutes to address the issue. The issue was fully stabilized and resolved by 7:20AM PDT. After a full analysis of the product update process, the engineering team has decided to change the update mechanism of the Keeper Desktop application to utilize a new distributed content delivery network which does not impact the other Keeper AWS infrastructure. This change will be implemented over the next 2 weeks and in preparation of the next Keeper Desktop application update.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2024-09-24 13:43 UTC | Critical incident: Keeper API website load time and API errors in US region (this change) | degradation |

All Keeper changes: [timeline](https://rewindex.com/company/keeper/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ic1bd3e45a91818 · Markdown: https://rewindex.com/change/ic1bd3e45a91818.md · JSON: https://rewindex.com/api/v1/changes/ic1bd3e45a91818 · Generated 2026-10-18T02:29:14.923Z