rewindex

CompaniesFly.ioIncidents

Fly.io: Slow image pulls delaying Machine starts in IAD

MinorStatus: resolvededitedStarted Resolved (1.5 h)4 updatesstatus.flyio.net (opens in new tab)

Affected components: IAD - Ashburn, Virginia (US)Deployments

Timeline of updates

  1. investigating

    We are investigating slow image pulls within Fly Machines in IAD. Newly created or updated Machines are taking longer to start, which may cause deployments to time out. Machines start successfully after the delay. Consider creating Machines in alternate regions, if possible, or running fly deploy --wait-timeout 10m to give Machines up to 10 minutes to prepare images and start.

  2. identified

    We have identified the cause of the issue and are working on a fix.

  3. monitoring

    We have implemented a fix and are monitoring results.

  4. resolved

    Image pull times in IAD have returned to normal.

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on status.flyio.net (opens in new tab).

Fly.io profileRecent Fly.io incidentsMarkdownJSON