# Harness: Slowness in Prod1 and Prod2 environment

- Impact (vendor’s label): minor
- Status: postmortem
- Started: 2026-09-08 21:52 UTC · Resolved: 2026-09-09 04:30 UTC · Duration: 6.6 h
- Updates: 10 · Affected components: Platform, Prod 2 - Platform, Prod 2 - Software Engineering Insights (SEI) / AI DLC Insights, Software Engineering Insights (SEI) / AI DLC Insights, Code Repository, Prod 2 - Code Repository, Custom Dashboards, Prod 2 - Custom Dashboards, FME, Prod 1 - FME, Continuous Integration Enterprise(CIE) - Self Hosted Runners, Prod 2 - Continuous Integration Enterprise(CIE) - Self Hosted Runners, FME, Prod 2 - FME, Infrastructure as Code Management (IaCM), Prod 2 - Infrastructure as Code Management (IaCM), Cloud Cost Management (CCM), Prod 2 - Cloud Cost Management (CCM), Feature Flags (FF), Prod 1 - Feature Flags (FF), Chaos Engineering, Prod 2 - Chaos Engineering, Feature Flags (FF), Prod 2 - Feature Flags (FF), Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Prod 2 - Continuous Integration Enterprise(CIE) - Windows Cloud Builds, AI Test Automation, Prod 2 - AI Test Automation, Cloud Cost Management (CCM), Prod 1 - Cloud Cost Management (CCM), Artifact Registry, Prod 2 - Artifact Registry, Continuous Delivery - Next Generation (CDNG), Prod 2 - Continuous Delivery - Next Generation (CDNG), Prod 2 - Service Reliability Management (SRM), Service Reliability Management (SRM), Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Prod 2 - Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Delivery (CD) - FirstGen - EOS, Prod 1 - Continuous Delivery (CD) - FirstGen - EOS, Prod 2 - Security Testing Orchestration (STO), Security Testing Orchestration (STO), Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Prod 2 - Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Continuous Delivery (CD) - FirstGen - EOS, Prod 2 - Continuous Delivery (CD) - FirstGen - EOS, Internal Developer Portal (IDP), Prod 2 - Internal Developer Portal (IDP), Continuous Error Tracking (CET), Prod 2 - Continuous Error Tracking (CET), Prod 2 - Software Supply Chain Assurance (SSCA), Software Supply Chain Assurance (SSCA), Continuous Error Tracking (CET), Prod 1 - Continuous Error Tracking (CET), Continuous Delivery - Next Generation (CDNG), Prod 1 - Continuous Delivery - Next Generation (CDNG), Chaos Engineering, Prod 1 - Chaos Engineering
- [Vendor status page: https://stspg.io/nq6pjsvq1481](https://stspg.io/nq6pjsvq1481)
- [Company: Harness](https://rewindex.com/company/harness.md)

## Timeline of updates

- **investigating** 2026-09-08 21:52 UTC: We are currently investigating this issue.
- **investigating** 2026-09-08 22:53 UTC: We are continuing to investigate this issue.
- **investigating** 2026-09-08 22:53 UTC: We  are continuing to investigate the issue.
- **investigating** 2026-09-08 23:31 UTC: We are continuing to investigate this issue.
- **identified** 2026-09-08 23:52 UTC: The issue has been identified and a fix is being implemented.
- **identified** 2026-09-09 00:29 UTC: We are continuing to work on a fix for this issue.
- **monitoring** 2026-09-09 00:36 UTC: A fix has been implemented and we are monitoring the results.
- **monitoring** 2026-09-09 00:39 UTC: A fix has been implemented and we are monitoring the results.
- **resolved** 2026-09-09 04:30 UTC: This incident has been resolved.
- **postmortem** 2026-09-09 23:23 UTC: \#\#\# Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.  \#\#\# Customer Impact \* Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. \* No data loss or corruption occurred. \#\#\# Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. \#\#\# Mitigation \* Disabled the capability across all environments \* Temporarily increased platform capacity to restore stability \#\#\# Next Steps To prevent recurrence, Harness will: 1. \*\*Permanently fix the capability\*\* by profiling and eliminating the sub-optimal code path and query 2. \*\*Improve detection\*\* by enhancing alerting for resource-intensive queries on high-frequency platform paths


---
Canonical: https://rewindex.com/company/harness/incidents/jsbrf6kx58tw · Markdown: https://rewindex.com/company/harness/incidents/jsbrf6kx58tw.md · JSON: https://rewindex.com/api/v1/companies/harness/incidents/jsbrf6kx58tw · Generated 2026-10-20T00:03:23.826Z