Changes / Upstash / Fly.io Upstash Redis Service Distruption
Upstash: Critical incident: Fly.io Upstash Redis Service Distruption
Summary
A critical Fly.io Upstash Redis degradation caused intermittent hangs and elevated error rates for a subset of instances. It started at 2026-05-12 09:49 UTC and resolved at 2026-05-12 13:48 UTC, lasting 3 h 59 min. Upstash implemented a fix; its postmortem reported Redis processes stalling inside a logging syscall.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 42%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Fly.io Upstash Redis Service Distruption
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2026-05-12 09:49 UTC
Before: not present
After: 2026-05-12 13:48 UTC
Before: not present
After: Some regions are experiencing connectivity issues due to an ongoing network problem. We are currently investigating
Before: not present
After: The issue has been identified and the fix is being implemented.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Fly.io Upstash Redis Service Distruption (this change) A critical Fly.io Upstash Redis degradation caused intermittent hangs and elevated error rates for a subset of instances. It started at 2026-05-12 09:49 UTC and resolved at 2026-05-12 13:48 UTC, lasting 3 h 59 min. Upstash implemented a fix; its postmortem reported Redis processes stalling inside a logging syscall. | Major incident |