Changes / Bitrise / ORD data center network outage
Bitrise: Major incident: ORD data center network outage
Summary
Bitrise reported a major ORD data center outage affecting Builds (CI), with possible failures or delays, starting 2026-07-14 16:40 UTC and resolved after 25 h 33 min. Dropped packets between nodes caused failed health checks and cluster-wide disruption; mitigation restored job scheduling, and network configuration changes were applied across all data centers.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: ORD data center network outage
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-07-14 16:40 UTC
Before: not present
After: 2026-07-15 18:13 UTC
Before: not present
After: ⚙️ Builds (CI)
Before: not present
After: We are currently investigating a network outage affecting our ORD data center. Builds running in this data center may have failed or been delayed during the outage. Our team identified an issue with Proxmox storage in the affected data center and is working on a fix.
We will provide a further update as soon as more information is available.
Before: not present
After: The network outage affecting our ORD data center has been mitigated. Builds running in this data center may have failed or been delayed during the outage window. Our team identified and addressed an issue with Proxmox storage in the affected data center, and job scheduling has resumed. We are continuing to monitor stability as we investigate the underlying root cause.
We will provide a further update as soon as more information is available.
Before: not present
After: This incident has been resolved. We experienced a network outage in our ORD data center caused by dropped packets between nodes, which led to failed health checks and a cluster-wide disruption. The outage was mitigated on July 14, and we've since applied network configuration changes across all our data centers to address the underlying cause and prevent recurrence. We apologize for the disruption it caused.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: ORD data center network outage (this change) Bitrise reported a major ORD data center outage affecting Builds (CI), with possible failures or delays, starting 2026-07-14 16:40 UTC and resolved after 25 h 33 min. Dropped packets between nodes caused failed health checks and cluster-wide disruption; mitigation restored job scheduling, and network configuration changes were applied across all data centers. | Major incident |