Companies/Harness/Incidents

Harness: The Prod3 & Prod1 environment is experiencing intermittent outages. We are currently investigating the issue.

Affected components: Infrastructure as Code Management (IaCM)Prod 1 - Infrastructure as Code Management (IaCM)Code RepositoryProd 1 - Code RepositoryContinuous Integration Enterprise(CIE) - Mac Cloud BuildsProd 3 - Continuous Integration Enterprise(CIE) - Mac Cloud BuildsInternal Developer Portal (IDP)Prod 1 - Internal Developer Portal (IDP)Prod 1 - Software Engineering Insights (SEI) /AI DLC InsightsSoftware Engineering Insights (SEI) /AI DLC InsightsContinuous Integration Enterprise(CIE) - Linux Cloud BuildsProd 1 - Continuous Integration Enterprise(CIE) - Linux Cloud BuildsProd 3 - Software Supply Chain Assurance (SSCA)Software Supply Chain Assurance (SSCA)Custom DashboardsProd 1 - Custom DashboardsFMEProd 1 - FMEContinuous Integration Enterprise(CIE) - Self Hosted RunnersProd 3 - Continuous Integration Enterprise(CIE) - Self Hosted RunnersFeature Flags (FF)Prod 1 - Feature Flags (FF)Continuous Integration Enterprise(CIE) - Self Hosted RunnersProd 1 - Continuous Integration Enterprise(CIE) - Self Hosted RunnersCloud Cost Management (CCM)Prod 1 - Cloud Cost Management (CCM)Cloud Cost Management (CCM)Prod 3 - Cloud Cost Management (CCM)PlatformProd 3 - PlatformProd 1 - Release ManagementRelease ManagementProd 3 - Software Engineering Insights (SEI) / AI DLC InsightsSoftware Engineering Insights (SEI) / AI DLC InsightsFeature Flags (FF)Prod 3 - Feature Flags (FF)Custom DashboardsProd 3 - Custom DashboardsContinuous Integration Enterprise(CIE) - Mac Cloud BuildsProd 1 - Continuous Integration Enterprise(CIE) - Mac Cloud BuildsContinuous Delivery - Next Generation (CDNG)Prod 3 - Continuous Delivery - Next Generation (CDNG)AI Test AutomationProd 1 - AI Test AutomationProd 1 - Service Reliability Management (SRM)Service Reliability Management (SRM)Prod 1 - Software Supply Chain Assurance (SSCA)Software Supply Chain Assurance (SSCA)Internal Developer Portal (IDP)Prod 3 - Internal Developer Portal (IDP)Continuous Integration Enterprise(CIE) - Windows Cloud BuildsProd 1 - Continuous Integration Enterprise(CIE) - Windows Cloud BuildsProd 3 - Security Testing Orchestration (STO)Security Testing Orchestration (STO)Continuous Delivery (CD) - FirstGen - EOSProd 1 - Continuous Delivery (CD) - FirstGen - EOSContinuous Integration Enterprise(CIE) - Windows Cloud BuildsProd 3 - Continuous Integration Enterprise(CIE) - Windows Cloud BuildsProd 3 - Service Reliability Management (SRM)Service Reliability Management (SRM)Continuous Delivery (CD) - FirstGen - EOSProd 3 - Continuous Delivery (CD) - FirstGen - EOSArtifact RegistryProd 1 - Artifact RegistryArtifact RegistryProd 3 - Artifact RegistryProd 1 - Security Testing Orchestration (STO)Security Testing Orchestration (STO)Continuous Integration Enterprise(CIE) - Linux Cloud BuildsProd 3 - Continuous Integration Enterprise(CIE) - Linux Cloud BuildsInfrastructure as Code Management (IaCM)Prod 3 - Infrastructure as Code Management (IaCM)Continuous Error Tracking (CET)Prod 3 - Continuous Error Tracking (CET)Chaos EngineeringProd 3 - Chaos EngineeringContinuous Error Tracking (CET)Prod 1 - Continuous Error Tracking (CET)PlatformProd 1 - PlatformContinuous Delivery - Next Generation (CDNG)Prod 1 - Continuous Delivery - Next Generation (CDNG)Code RepositoryProd 3 - Code RepositoryChaos EngineeringProd 1 - Chaos Engineering

Timeline of updates

1. investigating2026-07-27 06:42 UTC

   We are currently investigating this issue.

2. identified2026-07-27 06:45 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring2026-07-27 06:49 UTC

   A fix has been implemented and we are monitoring the results.

4. investigating2026-07-27 07:52 UTC

   We are currently investigating this issue.

5. investigating2026-07-27 08:13 UTC

   We are continuing to investigate this issue.

6. identified2026-07-27 09:04 UTC

   The issue has been identified and a fix is being implemented.

7. identified2026-07-27 09:05 UTC

   We are continuing to work on a fix for this issue.

8. monitoring2026-07-27 09:07 UTC

   A fix has been implemented and we are monitoring the results.

9. monitoring2026-07-27 09:07 UTC

   We are continuing to monitor for any further issues.

10. monitoring2026-07-27 09:09 UTC

    We are continuing to monitor for any further issues.

11. resolved2026-07-27 13:33 UTC

    This incident has been resolved.

12. postmortem2026-08-07 18:15 UTC

    # Summary During a recent production deployment, a defect in our internal deployment tooling caused two critical services   to run with incorrect, non-production configuration values in our production environment This led to a related set of four distinct symptoms: incorrect configuration behavior, intermittent login/access failures, a filestore access issue affecting one customer environment, and delayed pipeline status updates in the UI. ‌ We have identified and are implementing a permanent fix for the underlying configuration defect, and have already put in place resource and capacity changes that resolve the UI delay symptom. ‌ At no point during this incident were pipeline executions themselves lost, corrupted, or left in a stuck state. Where execution behavior was affected, it was limited to delays in status visibility, not in the underlying processing. # Incident Details ## Incorrect Production Configuration Values Applied Our engineering team confirmed a defect in the Service Manager deployment pipeline that caused certain production services to be deployed using configuration values intended for a different environment, rather than the correct production configuration. ‌ **Root Cause** ‌ The service responsible for fetching configuration overrides during deployment queries an internal API that returns a maximum of 1,000 results per request. The total number of services in the environment recently grew beyond that limit. As a result, any service beyond the first 1,000 returned was not included in the response, and the deployment pipeline silently fell back to default configuration values for those services. This is a confirmed pagination defect in the deployment tooling, not an issue with the configuration values themselves. ‌ **Resolution** ‌ Engineering has confirmed the mechanism and is implementing a permanent fix to remove this limit-related gap in the deployment pipeline. ## Intermittent Login / Access Failures During the Service Manager deployment referenced above, some users experienced intermittent login or access failures. Under normal operation, previously running instances should continue serving traffic without interruption while a new deployment is in progress. In this incident, that fallback behavior did not occur as expected, contributing to access failures during the deployment window. ## Filestore Access Issue A filestore access issue was identified that was specific to the Prod-3 environment and affected a single customer's environment. **Root Cause** This is related to an IAM / storage-bucket permission configuration on Service Manager, potentially triggered by rollback activity.  ## Delayed Pipeline Execution Status Updates in UI Some users observed that the pipeline execution graph in the UI was slow to refresh and did not reflect the latest status promptly. Importantly, this was a visibility delay only: there was no impact to actual pipeline executions, and no executions were stuck or failed as a result of this issue. **Root Cause** The pipeline execution graph relies on a message stream \(the orchestration log\) to receive status updates. During the incident window, consumer processing of this stream fell behind \(high consumer lag\), which delayed how quickly status updates reached the UI. This was caused by the fact that the underlying database was in the middle of a planned scaling operation at the same time, and a traffic spike during that window further exacerbated the delay. Users experienced this as apparent pipeline slowness, even though the underlying executions were running normally. **Resolution** We have increased resource capacity for the affected components to maintain more than 50% spare headroom going forward, reducing sensitivity to similar load spikes. This change has been implemented and is currently being validated as part of longer-term hardening for this part of the platform. # Impact Summary * Service Manager and License Manager ran with incorrect configuration values in the Prod-1 and Prod-3 environments. * Some users experienced intermittent login or access failures during the affected deployment window. * One customer environment in Prod-3 experienced a filestore access issue. * Users across affected environments saw delayed pipeline execution status updates in the UI; underlying pipeline executions continued to run correctly and were not lost, stuck, or corrupted. # Preventive Actions The following corrective and preventive actions have been identified. ‌ | **Corrective / Preventive Action** | | --- | | Correct the pagination limit in the configuration-lookup service so that all services are returned and evaluated, regardless of total count. | | Add safeguards so that a service which cannot retrieve its configuration fails safely \(e.g. alerts and blocks the deployment\) rather than silently falling back to non-production defaults. | | Increase Postgres and messaging-pipeline resource headroom \(target: greater than 50% spare capacity\) to reduce sensitivity to concurrent load and scaling events. | ‌ _We recognize the impact this incident had across multiple areas of the platform and appreciate your patience as we work through a complete resolution._

Changes Rewindex caught on this incident

• Jul 27Major incident: The Prod3 & Prod1 environment is experiencing intermittent outages. We are currently investigating the issue.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Harness profileRecent Harness incidentsMarkdownJSON