Changes / Fly.io / Platform Wide Rolling Host Reboots
Fly.io: Major incident: Platform Wide Rolling Host Reboots
Summary
Fly.io’s major fleet-wide host reboots mitigated PwnEscape (Linux Kernel CVE-2026-99999), making Machines, volumes and MPG instances unavailable for 15–20 minutes per host. The incident ran from 2026-10-16 16:22 UTC to 2026-10-17 05:15 UTC (12 h 52 min); it was resolved, though a small number of hosts remained degraded or offline.
Machine-written from the diff below (Sol; type: incident security). Check the diff before relying on it.
What changed
Before: not present
After: Platform Wide Rolling Host Reboots
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-10-16 16:22 UTC
Before: not present
After: 2026-10-17 05:15 UTC
Before: not present
After: We will be imminently applying rolling restarts across all hosts in our fleet for a critical maintenance. Customer Machines, volumes, and MPG cluster instances on each host will be unavailable while their host reboots.
Each host will be unavailable for 15 - 20 minutes during the restart, and multiple hosts in a region may be down at once. Users should expect a period of downtime or disruption for their apps, especially if they are running on a single machine.
We are aiming for the highest possible uptime during the rollout, while ensuring this is completed quickly.
We will provide a further update once we begin the restart process.
Before: not present
After: We are beginning the rolling restart process of all hosts, beginning with the YYZ region. Machines, volumes, and MPG instances on hosts being rebooted will be unavailable for approximately 15-20 minutes during the restart.
We will provide further updates as the rollout progresses.
Before: not present
After: Rolling host restarts are beginning in all regions.
Machines, volumes, and MPG instances will be unavailable for approximately 15-20 minutes when their hosts are restarted during this process.
Users should expect a period of disruption or downtime for their apps during this, especially if they are running on a single machine.
We will provide further updates as the rollout progresses.
Before: not present
After: Host restarts are continuing in phases across all regions. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions, however MPG users will likely see elevated primary <> standby switchovers during this period.
We will provide further updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions, with the first phase nearly complete. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions, with the second phase nearly complete - at this point, roughly 25% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 40% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 50% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 65% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 70% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 80% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 85% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: Host restarts are continuing in phases across all regions - at this point, roughly 90% of all hosts across the fleet have been restarted.
Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.
We
Before: not present
After: As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 90% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 2h30m.
Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 90% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 2h.
Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 1h40m.
Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 1h30m.
Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 30m.
Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise.
MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team.
We will continue to provide updates on progress as the rollout continues.
Before: not present
After: All host restarts have been completed and we are monitoring the fleet health.
A small number of hosts remain degraded or offline for apps and for MPG. Those impacted can see this status on the personalized status page viewable at https://fly.io/status
Before: not present
After: All host restarts have been completed and the platform is operating normally.
A small number of hosts remain degraded or offline for apps and for MPG. Those impacted can see further updates on the personalized status page viewable at https://fly.io/status
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Platform Wide Rolling Host Reboots (this change) Fly.io’s major fleet-wide host reboots mitigated PwnEscape (Linux Kernel CVE-2026-99999), making Machines, volumes and MPG instances unavailable for 15–20 minutes per host. The incident ran from 2026-10-16 16:22 UTC to 2026-10-17 05:15 UTC (12 h 52 min); it was resolved, though a small number of hosts remained degraded or offline. | Major incident |