# Svix: Major incident: Most routes 5xx in the US region [resolved]

**Summary:** _summary pending_
- Kind: Major incident · Label: major (vendor)
- Changed: 2023-03-11 20:35 UTC · Detected: 2026-10-19 18:53 UTC
- [Source: https://stspg.io/jxyqtbd7b2mz](https://stspg.io/jxyqtbd7b2mz)
- [Company: Svix](https://rewindex.com/company/svix.md)
- Versions: before (none: first archived) → after incident\_version:756/23w4yvk7rd6c@ff342fe5aeeb
- Change 1 of 1 for this incident

Incident page (full timeline): [Most routes 5xx in the US region \[resolved\]](https://rewindex.com/company/svix/incidents/23w4yvk7rd6c.md)
## What changed

**1. Added: Title**

```diff
+ Most routes 5xx in the US region [resolved]
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2023-03-11 20:35 UTC
```

**5. Added: Resolved time**

```diff
+ 2023-03-11 21:23 UTC
```

**6. Added: Affected components**

```diff
+ API
```

**7. Added: Update** (investigating) · 2023-03-11 19:38 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (investigating) · 2023-03-11 20:27 UTC

```diff
+ Still investigating.
```

**9. Added: Update** (investigating) · 2023-03-11 20:35 UTC

```diff
+ We are still investigating, but we put mitigating changes in place.
```

**10. Added: Update** (monitoring) · 2023-03-11 20:42 UTC

```diff
+ The service is back up. We are still monitoring, but everything is operational.
```

**11. Added: Update** (resolved) · 2023-03-11 21:23 UTC

```diff
+ The issue has been resolved, though we are still trying to locate the root cause. There were no deploys today (Saturday), so it's not due to any change on our end, and the activity doesn't look unusual. For whatever reason our API workers went from ~10% utilization (normal) to 100% in a short span of time. We are investigating with AWS. Update: after investigating with AWS for a few hours, neither they nor us are able to understand the reason for the memory usage jump though their and our metrics don't indicate any change in load, underlying systems, or anything like that. They've indicated that OOM can happen even if there hasn't been any indication to that in the AWS metrics.  We are still investigating.
```

**12. Added: Update** (postmortem) · 2023-03-12 21:36 UTC

```diff
+ We are still investigating, but here is our more complete update regarding this incident: [https://www.svix.com/blog/we-had-a-partial-outage/](https://www.svix.com/blog/we-had-a-partial-outage/)
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2023-03-11 20:35 UTC | Major incident: Most routes 5xx in the US region \[resolved\] (this change) | major |

All Svix changes: [timeline](https://rewindex.com/company/svix/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i7786293c56fa93 · Markdown: https://rewindex.com/change/i7786293c56fa93.md · JSON: https://rewindex.com/api/v1/changes/i7786293c56fa93 · Generated 2026-10-19T21:43:47.082Z