# Upstash: Major incident: QStash Service Disruption in the EU Region

**Summary:** Upstash reported a major QStash EU outage starting 2026-06-19 09:11 UTC; computed duration: 0 min. Its postmortem says a networking configuration change degraded outbound connectivity between 09:11 and 09:45 UTC, potentially failing initial message dispatches and causing some servers to use IPs outside the advertised outbound range. Upstash said the issue was resolved. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-06-19 09:11 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/5m39sbnstm49](https://stspg.io/5m39sbnstm49)
- [Company: Upstash](https://rewindex.com/company/upstash.md)
- Versions: before (none: first archived) → after incident\_version:171/4rshtsqjhn3c@356039da386a
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ QStash Service Disruption in the EU Region
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2026-06-19 09:11 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-06-19 09:11 UTC
```

**6. Added: Update** (resolved) · 2026-06-19 09:11 UTC

```diff
+ We identified an outbound networking issue affecting a subset of QStash server instances in the EU region between 2026-06-19 09:11 UTC and 2026-06-19 09:45 UTC. During this period, some requests dispatched by QStash may have failed. The issue has been resolved, and QStash is currently operating normally. We will publish a postmortem with additional details as soon as it is available. Update: We identified that one QStash server had been assigned an outbound IP address outside of the configured QStash IP range. This remained the case until 2026-06-19 11:24 UTC. Endpoints that restrict access based on QStash outbound IP allowlists may have rejected requests from this server during this period.
```

**7. Added: Update** (postmortem) · 2026-06-19 13:52 UTC

```diff
+ ## What happened A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until **11:24 UTC**; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers. QStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally. ## Customer impact Impact was limited to the EU region and to two effects within the window above: * **Delivery delays / failures** _\(09:11 – 09:45 UTC\)_ — messages dispatched through affected instances may have failed on their first attempt. Thanks to automatic retries with backoff, most were re-delivered once connectivity recovered; messages that exhausted their retry policy during the window followed their configured failure path \(e.g., DLQ / failure callback\). * **Allowlist rejections** _\(09:11 – 11:24 UTC\)_ — customers who restrict inbound traffic to QStash's outbound IP ranges may have seen requests from the affected servers rejected. Customers who do not enforce source-IP allowlisting were unaffected by this. ## Root cause A routine networking configuration change contained an error that: 1. disrupted outbound connectivity on the affected nodes, and 2. caused affected nodes to acquire outbound IPs outside the advertised range. The underlying gap was systemic: the change procedure had no automated validation gate to catch the faulty state before it reached production. ## What we've done * **Fail-fast pre-checks** — automated validation now halts the change procedure _before it runs_ if a configuration would degrade connectivity or violate expected state. * **Egress IP-range enforcement** — outbound IP assignments are now checked against the advertised QStash range, so a node can no longer come online with an unadvertised IP. * **Tighter post-change verification** — completion now confirms outbound reachability and egress-IP conformance across affected nodes. ## Customer action None required. All QStash traffic again originates from the advertised outbound IP ranges. We apologize for the disruption.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-06-19 09:11 UTC | Major incident: QStash Service Disruption in the EU Region (this change) | outage |

All Upstash changes: [timeline](https://rewindex.com/company/upstash/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i3bb8739cda696f · Markdown: https://rewindex.com/change/i3bb8739cda696f.md · JSON: https://rewindex.com/api/v1/changes/i3bb8739cda696f · Generated 2026-10-18T02:54:38.354Z