Changes / Fly.io / Upstash unavailability in FRA region
Fly.io: Major incident: Upstash unavailability in FRA region
Summary
A major Upstash outage started 2026-09-23 12:42 UTC and resolved 2026-09-24 07:22 UTC (18 h 39 min). Redis connection timeouts affected databases hosted in fra or gig and clients routed through either region, including databases elsewhere. Replicas failed to finish draining during a planned rolling upgrade and remained out of service; a fix was implemented.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
Major incidentLabel: outage (model (Jev), 89%)Changed Detected Source: status.flyio.net (opens in new tab)
Incident page: full timeline of updates →
What changed
Before: not present
After: Upstash unavailability in FRA region
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-09-23 12:42 UTC
Before: not present
After: 2026-09-24 07:22 UTC
Before: not present
After: We are investigating an issue affecting Upstash services in the fra region. Customers may experience connection failures or service unavailability. We are working with Upstash to identify the cause and restore service. We’ll provide an update as soon as we have more information.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: Impact
From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region.
What happened
During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out.
What we're changing
• We will be adding tooling to return a replica to service safely after maintenance.
• We will be adding safety checks to our upgrade process.
• We will make the upgrade procedure more resilient to errors like this one.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When | Change | Kind |
|---|---|---|
| Major incident: Upstash unavailability in FRA region (this change) A major Upstash outage started 2026-09-23 12:42 UTC and resolved 2026-09-24 07:22 UTC (18 h 39 min). Redis connection timeouts affected databases hosted in fra or gig and clients routed through either region, including databases elsewhere. Replicas failed to finish draining during a planned rolling upgrade and remained out of service; a fix was implemented. | Major incident |