Changes / Fly.io / State database issues in GRU

Fly.io: Major incident: State database issues in GRU

Summary

Fly.io reported a major GRU outage starting 2026-09-24 16:31 UTC, lasting 2 h 40 min and resolved at 2026-09-24 19:11 UTC. Apps continued running, but machine creation or updates and connections could fail. Fly.io linked the issues to a distributed state database migration, identified a bug, and implemented a fix.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 66%)Changed Detected Source: status.flyio.net (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: State database issues in GRU
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2026-09-24 16:31 UTC
AddedResolved time#5
Before: not present
After: 2026-09-24 19:11 UTC
AddedUpdateinvestigating#6
Before: not present
After: We are investigating issues in GRU related to a migration of our distributed state database. Apps continue to run, but creating or updating machines in GRU region may fail. Connections to apps in GRU, or connections from clients near GRU region, may fail at this time
AddedUpdateidentified#7
Before: not present
After: We have identified a bug and are working on a fix.
AddedUpdatemonitoring#8
Before: not present
After: A fix has been implemented and we are monitoring the results.
AddedUpdateresolved#9
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: State database issues in GRU (this change)
Fly.io reported a major GRU outage starting 2026-09-24 16:31 UTC, lasting 2 h 40 min and resolved at 2026-09-24 19:11 UTC. Apps continued running, but machine creation or updates and connections could fail. Fly.io linked the issues to a distributed state database migration, identified a bug, and implemented a fix.
Major incident

All Fly.io changes → · Fly.io profile · Markdown · JSON