# Bitrise: Major incident: ORD data center network outage

**Summary:** Bitrise reported a major ORD data center outage affecting Builds (CI), with possible failures or delays, starting 2026-07-14 16:40 UTC and resolved after 25 h 33 min. Dropped packets between nodes caused failed health checks and cluster-wide disruption; mitigation restored job scheduling, and network configuration changes were applied across all data centers. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-07-14 16:40 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/pstd6w4q7gmv](https://stspg.io/pstd6w4q7gmv)
- [Company: Bitrise](https://rewindex.com/company/bitrise.md)
- Versions: before (none: first archived) → after incident\_version:177/rnk6slc2xvhm@8b93e7a66a76
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ ORD data center network outage
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-07-14 16:40 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-07-15 18:13 UTC
```

**6. Added: Affected components**

```diff
+ ⚙️ Builds (CI)
```

**7. Added: Update** (investigating) · 2026-07-14 16:40 UTC

```diff
+ We are currently investigating a network outage affecting our ORD data center. Builds running in this data center may have failed or been delayed during the outage. Our team identified an issue with Proxmox storage in the affected data center and is working on a fix.  We will provide a further update as soon as more information is available.
```

**8. Added: Update** (monitoring) · 2026-07-14 18:18 UTC

```diff
+ The network outage affecting our ORD data center has been mitigated. Builds running in this data center may have failed or been delayed during the outage window. Our team identified and addressed an issue with Proxmox storage in the affected data center, and job scheduling has resumed. We are continuing to monitor stability as we investigate the underlying root cause. We will provide a further update as soon as more information is available.
```

**9. Added: Update** (resolved) · 2026-07-15 18:13 UTC

```diff
+ This incident has been resolved. We experienced a network outage in our ORD data center caused by dropped packets between nodes, which led to failed health checks and a cluster-wide disruption. The outage was mitigated on July 14, and we've since applied network configuration changes across all our data centers to address the underlying cause and prevent recurrence. We apologize for the disruption it caused.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-07-14 16:40 UTC | Major incident: ORD data center network outage (this change) | outage |

All Bitrise changes: [timeline](https://rewindex.com/company/bitrise/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ibd6f1ab0e9b391 · Markdown: https://rewindex.com/change/ibd6f1ab0e9b391.md · JSON: https://rewindex.com/api/v1/changes/ibd6f1ab0e9b391 · Generated 2026-10-18T02:29:46.763Z