# Harness: Intermittent External Network Connectivity Issues Affecting Build VMs

- Impact (vendor’s label): minor
- Status: postmortem
- Started: 2026-07-30 05:59 UTC · Resolved: 2026-07-31 03:22 UTC · Duration: 21 h
- Updates: 5 · Affected components: Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 1 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 2 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 3 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 4 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod Eu 1 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds
- [Vendor status page: https://stspg.io/qdjcv8bw5w5x](https://stspg.io/qdjcv8bw5w5x)
- [Company: Harness](https://rewindex.com/company/harness.md)

## Timeline of updates

- **investigating** 2026-07-30 05:59 UTC: Summary - We are intermittently facing network connectivity issues with our Build VM's unable to connect to external resources. We are currently investigating the issue.
- **monitoring** 2026-07-30 08:07 UTC: A fix has been implemented and we are monitoring the results.
- **monitoring** 2026-07-30 08:07 UTC: We are continuing to monitor for any further issues.
- **resolved** 2026-07-31 03:22 UTC: This incident has been resolved.
- **postmortem** 2026-08-07 19:32 UTC: \#\# Summary CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways. \#\# Impact \* CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services \\(e.g., GitHub, Bitbucket\\) over our outbound network gateways. \* The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume. \* No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue. \* us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue. \#\# Root Cause ‌ Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details \\(source/destination address and port\\). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways ‌ \#\# Action Items To prevent such issues from happening again Harness will, Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..


---
Canonical: https://rewindex.com/company/harness/incidents/6x15cj3h1pkr · Markdown: https://rewindex.com/company/harness/incidents/6x15cj3h1pkr.md · JSON: https://rewindex.com/api/v1/companies/harness/incidents/6x15cj3h1pkr · Generated 2026-10-20T02:21:33.306Z