Companies / Fly.io / incidents

Fly.io: Platform Wide Rolling Host Reboots

Impact: majorStatus: resolvedStarted Resolved (13 h)20 updatesSource: status.flyio.net (opens in new tab)
Summary

Fly.io’s major fleet-wide host reboots mitigated PwnEscape (Linux Kernel CVE-2026-99999), making Machines, volumes and MPG instances unavailable for 15–20 minutes per host. The incident ran from 2026-10-16 16:22 UTC to 2026-10-17 05:15 UTC (12 h 52 min); it was resolved, though a small number of hosts remained degraded or offline.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: ORD - Chicago, Illinois (US)ORD - Chicago, Illinois (US)Management Plane - FRAManagement Plane - NRTARN - Stockholm, SwedenARN - Stockholm, SwedenLAX - Los Angeles, California (US)LAX - Los Angeles, California (US)SYD - Sydney, AustraliaSYD - Sydney, AustraliaManagement Plane - ORDDFW - Dallas, Texas (US)DFW - Dallas, Texas (US)Management Plane - LAXJNB - Johannesburg, South AfricaJNB - Johannesburg, South AfricaIAD - Ashburn, Virginia (US)IAD - Ashburn, Virginia (US)Management Plane - SYDEWR - Secaucus, NJ (US)EWR - Secaucus, NJ (US)CDG - Paris, FranceCDG - Paris, FranceFRA - Frankfurt, GermanyFRA - Frankfurt, GermanyBOM - Mumbai, IndiaBOM - Mumbai, IndiaGRU - Sao Paulo, BrazilGRU - Sao Paulo, BrazilAMS - Amsterdam, NetherlandsAMS - Amsterdam, NetherlandsManagement Plane - AMSManagement Plane - GRUManagement Plane - IADManagement Plane - SJCLHR - London, United KingdomLHR - London, United KingdomNRT - Tokyo, JapanNRT - Tokyo, JapanYYZ - Toronto, CanadaYYZ - Toronto, CanadaManagement Plane - YYZManagement Plane - YYZManagement Plane - LHRSIN - SingaporeSIN - SingaporeSJC - San Jose, California (US)SJC - San Jose, California (US)Management Plane - SINCustomer ApplicationsCustomer Applications

Timeline of updates

  1. identified

    We will be imminently applying rolling restarts across all hosts in our fleet for a critical maintenance. Customer Machines, volumes, and MPG cluster instances on each host will be unavailable while their host reboots. Each host will be unavailable for 15 - 20 minutes during the restart, and multiple hosts in a region may be down at once. Users should expect a period of downtime or disruption for their apps, especially if they are running on a single machine. We are aiming for the highest possible uptime during the rollout, while ensuring this is completed quickly. We will provide a further update once we begin the restart process.

  2. identified

    We are beginning the rolling restart process of all hosts, beginning with the YYZ region. Machines, volumes, and MPG instances on hosts being rebooted will be unavailable for approximately 15-20 minutes during the restart. We will provide further updates as the rollout progresses.

  3. identified

    Rolling host restarts are beginning in all regions. Machines, volumes, and MPG instances will be unavailable for approximately 15-20 minutes when their hosts are restarted during this process. Users should expect a period of disruption or downtime for their apps during this, especially if they are running on a single machine. We will provide further updates as the rollout progresses.

  4. identified

    Host restarts are continuing in phases across all regions. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions, however MPG users will likely see elevated primary <> standby switchovers during this period. We will provide further updates on progress as the rollout continues.

  5. identified

    Host restarts are continuing in phases across all regions, with the first phase nearly complete. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  6. identified

    Host restarts are continuing in phases across all regions, with the second phase nearly complete - at this point, roughly 25% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  7. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 40% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  8. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 50% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  9. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 65% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  10. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 70% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  11. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 80% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  12. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 85% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We will continue to provide updates on progress as the rollout continues.

  13. identified

    Host restarts are continuing in phases across all regions - at this point, roughly 90% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period. We

  14. identified

    As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 90% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 2h30m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.

  15. identified

    As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 90% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 2h. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.

  16. identified

    As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 1h40m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.

  17. identified

    As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 1h30m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.

  18. identified

    As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 30m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.

  19. monitoring

    All host restarts have been completed and we are monitoring the fleet health. A small number of hosts remain degraded or offline for apps and for MPG. Those impacted can see this status on the personalized status page viewable at https://fly.io/status

  20. resolved

    All host restarts have been completed and the platform is operating normally. A small number of hosts remain degraded or offline for apps and for MPG. Those impacted can see further updates on the personalized status page viewable at https://fly.io/status

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on status.flyio.net (opens in new tab). Fly.io profile · Recent Fly.io incidents · Markdown · JSON