Harness: Slowness in Prod1 and Prod2 environment
Affected components: PlatformProd 2 - PlatformProd 2 - Software Engineering Insights (SEI) / AI DLC InsightsSoftware Engineering Insights (SEI) / AI DLC InsightsCode RepositoryProd 2 - Code RepositoryCustom DashboardsProd 2 - Custom DashboardsFMEProd 1 - FMEContinuous Integration Enterprise(CIE) - Self Hosted RunnersProd 2 - Continuous Integration Enterprise(CIE) - Self Hosted RunnersFMEProd 2 - FMEInfrastructure as Code Management (IaCM)Prod 2 - Infrastructure as Code Management (IaCM)Cloud Cost Management (CCM)Prod 2 - Cloud Cost Management (CCM)Feature Flags (FF)Prod 1 - Feature Flags (FF)Chaos EngineeringProd 2 - Chaos EngineeringFeature Flags (FF)Prod 2 - Feature Flags (FF)Continuous Integration Enterprise(CIE) - Windows Cloud BuildsProd 2 - Continuous Integration Enterprise(CIE) - Windows Cloud BuildsAI Test AutomationProd 2 - AI Test AutomationCloud Cost Management (CCM)Prod 1 - Cloud Cost Management (CCM)Artifact RegistryProd 2 - Artifact RegistryContinuous Delivery - Next Generation (CDNG)Prod 2 - Continuous Delivery - Next Generation (CDNG)Prod 2 - Service Reliability Management (SRM)Service Reliability Management (SRM)Continuous Integration Enterprise(CIE) - Mac Cloud BuildsProd 2 - Continuous Integration Enterprise(CIE) - Mac Cloud BuildsContinuous Delivery (CD) - FirstGen - EOSProd 1 - Continuous Delivery (CD) - FirstGen - EOSProd 2 - Security Testing Orchestration (STO)Security Testing Orchestration (STO)Continuous Integration Enterprise(CIE) - Linux Cloud BuildsProd 2 - Continuous Integration Enterprise(CIE) - Linux Cloud BuildsContinuous Delivery (CD) - FirstGen - EOSProd 2 - Continuous Delivery (CD) - FirstGen - EOSInternal Developer Portal (IDP)Prod 2 - Internal Developer Portal (IDP)Continuous Error Tracking (CET)Prod 2 - Continuous Error Tracking (CET)Prod 2 - Software Supply Chain Assurance (SSCA)Software Supply Chain Assurance (SSCA)Continuous Error Tracking (CET)Prod 1 - Continuous Error Tracking (CET)Continuous Delivery - Next Generation (CDNG)Prod 1 - Continuous Delivery - Next Generation (CDNG)Chaos EngineeringProd 1 - Chaos Engineering
Timeline of updates
- investigating
We are currently investigating this issue.
- investigating
We are continuing to investigate this issue.
- investigating
We are continuing to investigate the issue.
- investigating
We are continuing to investigate this issue.
- identified
The issue has been identified and a fix is being implemented.
- identified
We are continuing to work on a fix for this issue.
- monitoring
A fix has been implemented and we are monitoring the results.
- monitoring
A fix has been implemented and we are monitoring the results.
- resolved
This incident has been resolved.
- postmortem
### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period. ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).