# Fly.io: Major incident: Platform Wide Rolling Host Reboots

**Summary:** Fly.io’s major fleet-wide host reboots mitigated PwnEscape (Linux Kernel CVE-2026-99999), making Machines, volumes and MPG instances unavailable for 15–20 minutes per host. The incident ran from 2026-10-16 16:22 UTC to 2026-10-17 05:15 UTC (12 h 52 min); it was resolved, though a small number of hosts remained degraded or offline. _(machine-written, sol)_
- Kind: Major incident · Label: security (model (Jev))
- Changed: 2026-10-16 16:22 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://status.flyio.net/incidents/01M52RAWMVC0NKEBRDKC3VXT3K](https://status.flyio.net/incidents/01M52RAWMVC0NKEBRDKC3VXT3K)
- [Company: Fly.io](https://rewindex.com/company/fly-io.md)
- Versions: before (none: first archived) → after incident\_version:186/01M52RAWMVC0NKEBRDKC3VXT3K@22912a75f7f4
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Platform Wide Rolling Host Reboots
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-10-16 16:22 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-10-17 05:15 UTC
```

**6. Added: Update** (identified) · 2026-10-16 16:22 UTC

```diff
+ We will be imminently applying rolling restarts across all hosts in our fleet for a critical maintenance. Customer Machines, volumes, and MPG cluster instances on each host will be unavailable while their host reboots. Each host will be unavailable for 15 - 20 minutes during the restart, and multiple hosts in a region may be down at once. Users should expect a period of downtime or disruption for their apps, especially if they are running on a single machine. We are aiming for the highest possible uptime during the rollout, while ensuring this is completed quickly. We will provide a further update once we begin the restart process.
```

**7. Added: Update** (identified) · 2026-10-16 16:50 UTC

```diff
+ We are beginning the rolling restart process of all hosts, beginning with the YYZ region. Machines, volumes, and MPG instances on hosts being rebooted will be unavailable for approximately 15-20 minutes during the restart. We will provide further updates as the rollout progresses.
```

**8. Added: Update** (identified) · 2026-10-16 16:59 UTC

```diff
+ Rolling host restarts are beginning in all regions.  Machines, volumes, and MPG instances will be unavailable for approximately 15-20 minutes when their hosts are restarted during this process. Users should expect a period of disruption or downtime for their apps during this, especially if they are running on a single machine. We will provide further updates as the rollout progresses.
```

**9. Added: Update** (identified) · 2026-10-16 18:01 UTC

```diff
+ Host restarts are continuing in phases across all regions. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete.  Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions, however MPG users will likely see elevated primary <> standby switchovers during this period.  We will provide further updates on progress as the rollout continues.
```

**10. Added: Update** (identified) · 2026-10-16 18:35 UTC

```diff
+ Host restarts are continuing in phases across all regions, with the first phase nearly complete. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete.  Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**11. Added: Update** (identified) · 2026-10-16 19:10 UTC

```diff
+ Host restarts are continuing in phases across all regions, with the second phase nearly complete - at this point, roughly 25% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete.  Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**12. Added: Update** (identified) · 2026-10-16 20:00 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 40% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**13. Added: Update** (identified) · 2026-10-16 20:41 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 50% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**14. Added: Update** (identified) · 2026-10-16 21:11 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 65% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**15. Added: Update** (identified) · 2026-10-16 21:41 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 70% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**16. Added: Update** (identified) · 2026-10-16 22:27 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 80% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**17. Added: Update** (identified) · 2026-10-16 23:15 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 85% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We will continue to provide updates on progress as the rollout continues.
```

**18. Added: Update** (identified) · 2026-10-17 00:05 UTC

```diff
+ Host restarts are continuing in phases across all regions - at this point, roughly 90% of all hosts across the fleet have been restarted. Only a subset of hosts in each region will be down at each time. We expect this process to take multiple hours to complete in full, with larger regions taking the longest to complete. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. Rollout phases were scheduled to minimize impact to Managed Postgres Clusters within those regions; however, MPG users will likely see elevated primary <> standby switchovers during this period.  We
```

**19. Added: Update** (identified) · 2026-10-17 00:45 UTC

```diff
+ As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 90% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 2h30m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.
```

**20. Added: Update** (identified) · 2026-10-17 01:11 UTC

```diff
+ As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 90% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 2h. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.
```

**21. Added: Update** (identified) · 2026-10-17 01:43 UTC

```diff
+ As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 1h40m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.
```

**22. Added: Update** (identified) · 2026-10-17 02:17 UTC

```diff
+ As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 1h30m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.
```

**23. Added: Update** (identified) · 2026-10-17 02:41 UTC

```diff
+ As a mitigation measure for PwnEscape (Linux Kernel CVE-2026-99999), host restarts are continuing in phases across all regions. At this point, more than 95% of the fleet has been restarted. The remaining hosts are expected to be restarted over the next 30m. Only a subset of hosts in each region will be down at each time. In general, Fly Machines should return to whatever state they were in prior to the host reboot, unless the Machine's restart policy says otherwise. MPG users will likely see elevated primary <> standby switchovers during this period. Impacted MPG clusters are being attended to by the team. We will continue to provide updates on progress as the rollout continues.
```

**24. Added: Update** (monitoring) · 2026-10-17 03:48 UTC

```diff
+ All host restarts have been completed and we are monitoring the fleet health. A small number of hosts remain degraded or offline for apps and for MPG. Those impacted can see this status on the personalized status page viewable at https://fly.io/status
```

**25. Added: Update** (resolved) · 2026-10-17 05:15 UTC

```diff
+ All host restarts have been completed and the platform is operating normally. A small number of hosts remain degraded or offline for apps and for MPG. Those impacted can see further updates on the personalized status page viewable at https://fly.io/status
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-10-16 16:22 UTC | Major incident: Platform Wide Rolling Host Reboots (this change) | security |

All Fly.io changes: [timeline](https://rewindex.com/company/fly-io/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i4302052fa544f8 · Markdown: https://rewindex.com/change/i4302052fa544f8.md · JSON: https://rewindex.com/api/v1/changes/i4302052fa544f8 · Generated 2026-10-17T23:13:47.639Z