ChangesSvixMost routes 5xx in the US region [resolved]
Svix: Major incident: Most routes 5xx in the US region [resolved]
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Svix removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Most routes 5xx in the US region [resolved]
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-03-11 20:35 UTC
Before: not present
After: 2023-03-11 21:23 UTC
Before: not present
After: API
Before: not present
After: We are currently investigating this issue.
Before: not present
After: Still investigating.
Before: not present
After: We are still investigating, but we put mitigating changes in place.
Before: not present
After: The service is back up. We are still monitoring, but everything is operational.
Before: not present
After: The issue has been resolved, though we are still trying to locate the root cause.
There were no deploys today (Saturday), so it's not due to any change on our end, and the activity doesn't look unusual.
For whatever reason our API workers went from ~10% utilization (normal) to 100% in a short span of time. We are investigating with AWS.
Update: after investigating with AWS for a few hours, neither they nor us are able to understand the reason for the memory usage jump though their and our metrics don't indicate any change in load, underlying systems, or anything like that. They've indicated that OOM can happen even if there hasn't been any indication to that in the AWS metrics.
We are still investigating.
Before: not present
After: We are still investigating, but here is our more complete update regarding this incident:
[https://www.svix.com/blog/we-had-a-partial-outage/](https://www.svix.com/blog/we-had-a-partial-outage/)
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Saturday, March 11, 2023
1 incident change
- MajorMajor incident
Most routes 5xx in the US region [resolved]this change