Changes / Upstash / Fly.io Upstash Redis service distruption (FRA region)
Upstash: Major incident: Fly.io Upstash Redis service distruption (FRA region)
Summary
Upstash reported a major degradation affecting some Fly.io Redis databases in FRA, with increased latency or timeouts. It started at 2026-05-11 15:05 UTC and resolved at 2026-05-11 18:22 UTC, lasting 3 h 16 min. The postmortem described Redis processes stalling inside a logging syscall; the supplied root-cause explanation is truncated.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 56%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Fly.io Upstash Redis service distruption (FRA region)
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-05-11 15:05 UTC
Before: not present
After: 2026-05-11 18:22 UTC
Before: not present
After: Some databases may experience increased latency or timeouts in Fly.io’s FRA region.
Before: not present
After: We are continuing to investigate the issue.
Before: not present
After: We are working with Fly team to investigate the root cause.
Before: not present
After: The incident has been resolved. We are working with Fly team on RCA.
Before: not present
After: On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Fly.io Upstash Redis service distruption (FRA region) (this change) Upstash reported a major degradation affecting some Fly.io Redis databases in FRA, with increased latency or timeouts. It started at 2026-05-11 15:05 UTC and resolved at 2026-05-11 18:22 UTC, lasting 3 h 16 min. The postmortem described Redis processes stalling inside a logging syscall; the supplied root-cause explanation is truncated. | Major incident |