Struck in red is what Harness removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Summary
On October 19, 2026, between approximately 12:10 AM and 12:55 AM PDT, customers on Prod4 experienced errors and intermittent unavailability of Harness Pipelines services. Heavy pipeline executions drove a sharp rise in CPU and memory use on the pipeline service, and for a short period all service instances were unavailable. Harness engineering scaled the service both vertically and horizontally, which restored service. A fix for the underlying behavior is being rolled out.
Root Cause
On October 19th, high number of concurrent executions with very large execution graphs drove a sharp increase in cluster resources. The service stores and retrieves this full execution graph for visualization purposes and further progressing the pipelines to next states. Our auto-scalaers on Prod4 took longer than usual for scaling resources due to a sharp rise in CPU and memory.
As a part of the fix, the execution graph update will now process graph changes incrementally instead of the full graph. It is being rolled out in a phased manner and will be completely rolled out at the earliest.
Impact
Affected users: Customers on Prod4.
Symptom: unavailability of pipeline functionality in the Harness platform.
Data: No data loss.
Suggested workaround: None required from customers. Harness engineering resolved the issue.
Duration of impact: Approximately 20 minutes of full unavailability and degraded performance for 25 minutes. Harness continued monitoring after recovery and confirmed platform stability with customers.
Remediation
Immediate
Harness engineering scaled the pipeline service both vertically (more resources per instance) and horizontally (more instances), and removed instances in an unhealthy state. The instance count stabilized and normal service was restored. The system will stay scaled up till the fix is completely rolled out.
Permanent fix and Action Items
To prevent such issues from happening again, Harness is
Tuning the auto heal/scaling process so that impact is mimimal to none.
Optimizing the execution graph flow so that large graphs can be process faster.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.