Companies/Harness/Incidents

Harness: Degraded CI performance

Affected components: Continuous Integration Enterprise(CIE) - Linux Cloud BuildsProd 1 - Continuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Integration Enterprise(CIE) - Self Hosted RunnersProd 1 - Continuous Integration Enterprise(CIE) - Self Hosted RunnersContinuous Integration Enterprise(CIE) - Mac Cloud BuildsProd 1 - Continuous Integration Enterprise(CIE) - Mac Cloud BuildsContinuous Integration Enterprise(CIE) - Windows Cloud BuildsProd 1 - Continuous Integration Enterprise(CIE) - Windows Cloud Builds

Timeline of updates

1. investigating2026-07-17 17:16 UTC

   We are currently investigating this issue.

2. identified2026-07-17 17:34 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring2026-07-17 18:12 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved2026-07-17 21:56 UTC

   This incident has been resolved.

5. postmortem2026-08-04 02:48 UTC

   # Summary On 17 July 2026, following a routine code deployment, customers on older delegate versions \(858xx and below\) began experiencing delayed CI builds on Harness Cloud-hosted builds using our global build-queueing capability. Affected builds experienced an unexpected pause of up to approximately 8 minutes at the "waiting for infrastructure" stage before continuing, rather than proceeding within the expected sub-second time. Overall build slowness was intermittent. # Impact * All CI builds were potentially subject to delay; impact was most pronounced for builds on Harness Cloud-hosted infrastructure using the global build-queueing feature. * Affected builds experienced an unexplained pause of up to approximately 8 minutes before continuing, followed by a slower "cold start" since a pre-reserved compute slot was not available — this presented to users as slow builds rather than build failures. * Accounts running on newer delegate versions \(858xx and above\) were not impacted. * No builds failed outright as a direct result of this issue, and no data was lost. # Root Cause The root cause was an internal code change that inadvertently broke how a specific build-queueing record was read back from our database once builds that had already been queued under the previous version of the code encountered the newly deployed version. We resolved the immediate impact by cleaning up the affected records and reverting the underlying code change, and we are implementing several safeguards to prevent this class of issue from recurring. ‌ # Next Steps We assess the risk of a similar recurrence as low as the following actions are being understaken.The specific code path that caused this incident has already been reverted, and we are implementing structural safeguards so that this general class of issue cannot recur, regardless of where in the codebase it might otherwise occur. | **Corrective / Preventive Action** | | --- | | Add explicit, stable identifiers to all internal data classes that get stored in our database, so that future internal code reorganizations cannot break the system's ability to read back previously stored records. | | Introduce rollback and backward-compatibility testing in our pre-production environment, specifically designed to catch this class of issue before it reaches production. |

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Harness profileRecent Harness incidentsMarkdownJSON