Companies / Upstash / incidents
Upstash: Fly.io Upstash Redis Service Distruption
A critical Fly.io Upstash Redis degradation caused intermittent hangs and elevated error rates for a subset of instances. It started at 2026-05-12 09:49 UTC and resolved at 2026-05-12 13:48 UTC, lasting 3 h 59 min. Upstash implemented a fix; its postmortem reported Redis processes stalling inside a logging syscall.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Timeline of updates
- investigating
Some regions are experiencing connectivity issues due to an ongoing network problem. We are currently investigating
- identified
The issue has been identified and the fix is being implemented.
- monitoring
A fix has been implemented and we are monitoring the results.
- resolved
This incident has been resolved.
- postmortem
On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.
Rewindex changes for this incident
- Critical incident: Fly.io Upstash Redis Service Distruption 2026-05-12 09:49 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Upstash profile · Recent Upstash incidents · Markdown · JSON