# Upstash: Critical incident: Degraded Performance

**Summary:** Upstash reported critical degraded performance affecting regional and Redis Global components from 2025-06-11 06:51 UTC to 2025-06-11 10:02 UTC (3 h 10 min). OS maintenance updates changed networking components and triggered restarts, causing EC2 nodes to fail health checks and disrupting high availability; a fix was implemented at 06-11 09:20 UTC. _(machine-written, sol)_
- Kind: Major incident · Label: degradation (model (Jev))
- Changed: 2025-06-11 06:51 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/rywr85m7ccvt](https://stspg.io/rywr85m7ccvt)
- [Company: Upstash](https://rewindex.com/company/upstash.md)
- Versions: before (none: first archived) → after incident\_version:171/kj6y893b91pq@595342171094
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Degraded Performance
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2025-06-11 06:51 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-06-11 10:02 UTC
```

**6. Added: Affected components**

```diff
+ Frankfurt, Germany (eu-central-1), Ireland (eu-west-1), London, UK (eu-west-2), Mumbai, India (ap-south-1), N. California, USA (us-west-1), N. Virginia, USA (us-east-1), Ohio, USA (us-east-2), Oregon, USA (us-west-2), Redis Global - Frankfurt, Germany (eu-central-1), Redis Global - Ireland (eu-west-1), Redis Global - London, UK (eu-west-2), Redis Global - Mumbai, India (ap-south-1), Redis Global - N. California, USA (us-west-1), Redis Global - N. Virginia, USA (us-east-1), Redis Global - Ohio, USA (us-east-2), Redis Global - Oregon, USA (us-west-2), Redis Global - São Paulo, Brazil (sa-east-1), Redis Global - Singapore (ap-southeast-1), Redis Global - Sydney, Australia (ap-southeast-2), Redis Global - Tokyo, Japan (ap-northeast-1), São Paulo, Brazil (sa-east-1), Singapore (ap-southeast-1), Sydney, Australia (ap-southeast-2), Tokyo, Japan (ap-northeast-1)
```

**7. Added: Update** (investigating) · 2025-06-11 06:51 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (investigating) · 2025-06-11 08:24 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (monitoring) · 2025-06-11 09:20 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**10. Added: Update** (resolved) · 2025-06-11 10:02 UTC

```diff
+ This incident has been resolved.
```

**11. Added: Update** (postmortem) · 2025-06-11 11:49 UTC

```diff
+ A routine system maintenance operation at the OS level led to the application of system updates across multiple EC2 instances in our clusters in several AWS regions. These updates included changes to networking components, which inadvertently triggered restarts.  As a result, several EC2 nodes failed health checks and temporarily dropped out of the cluster, disrupting high availability and causing partial connectivity issues for some clients and operations.  We have since reproduced the issue in a controlled environment and verified the root cause. To prevent a recurrence, we are updating our node maintenance strategy to ensure greater control over the timing and impact of system-level changes and excluding networking components from automated upgrades.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-06-11 06:51 UTC | Critical incident: Degraded Performance (this change) | degradation |

All Upstash changes: [timeline](https://rewindex.com/company/upstash/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i4aadd7f796a451 · Markdown: https://rewindex.com/change/i4aadd7f796a451.md · JSON: https://rewindex.com/api/v1/changes/i4aadd7f796a451 · Generated 2026-10-18T02:55:00.411Z