# Harness: CI Linux Cloud VM Networking issues - intermittent

- Impact (vendor’s label): minor
- Status: postmortem
- Started: 2026-09-29 18:45 UTC · Resolved: 2026-09-29 22:01 UTC · Duration: 3.3 h
- Updates: 4 · Affected components: Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 1 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 2 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds
- [Vendor status page: https://stspg.io/3ldn66vdtbzp](https://stspg.io/3ldn66vdtbzp)
- [Company: Harness](https://rewindex.com/company/harness.md)

## Timeline of updates

- **investigating** 2026-09-29 18:45 UTC: We are currently investigating this issue.
- **monitoring** 2026-09-29 21:49 UTC: A fix has been implemented and we are monitoring the results.
- **resolved** 2026-09-29 22:01 UTC: This incident has been resolved.
- **postmortem** 2026-10-02 22:40 UTC: \#\#\# Summary Between approximately \*\*11:37 AM and 3:01 PM PDT \\(18:37–22:01 UTC\\) on September 29, 2026\*\*, customers using Harness CI self-hosted Linux cloud runners in \*\*prod1/prod2 \\(us-west1\\)\*\* experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code.  \*\*Root Cause\*\* This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there. \#\#\# Impact \* Customers running self-hosted Linux CI builds in us-west1 saw \*\*intermittent build/pipeline failures\*\*, not a full outage. \* Observed customer-facing errors included Git fetch timeouts, e.g. \`Failed to connect to github.com port 443: Connection timed out\`  \*\*Remediation\*\* \* \*\*Immediate:\*\* Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 \\(partially limited by central1 capacity\\); restarted the affected Cloud NAT gateways, which stabilized pipeline networking. \*\*Action Items\*\* 1. Secure additional NAT gateway/IP capacity \\(In progress\\) 2. Improve NAT saturation detection and alerting ahead of customer impact. 3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.


---
Canonical: https://rewindex.com/company/harness/incidents/56bz1nnzmrbx · Markdown: https://rewindex.com/company/harness/incidents/56bz1nnzmrbx.md · JSON: https://rewindex.com/api/v1/companies/harness/incidents/56bz1nnzmrbx · Generated 2026-10-19T23:16:43.677Z