Struck in red is what Harness removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: 1. Summary
On October 27, 2026, starting around 6:35 AM, customers running CI pipelines on self-managed VM infrastructure (a legacy Delegate-based VM runner configuration) in Prod1 began experiencing pipeline failures caused by tasks being routed to Delegates that could not actually execute them. The issue affected a small number of customers using this specific, less common configuration (VM-based CI without Delegate tags configured). It was introduced by a backend deployment and was mitigated by rolling that deployment back; a permanent fix followed.
2. Impact
CI pipelines using VM-based execution without Delegate tags configured began intermittently failing to run on affected Delegates.
Other pipeline types and configurations were not affected.
Impact was limited to a small number of customers in Prod1 using this specific configuration.
The partial outage lasted approximately 8 hours before being resolved via rollback. Affected customers were given a configuration-based workaround (setting Delegate tags) during the incident.
3. Root Cause
Our Delegate platform uses a feature called Capability Checks to determine whether a given Delegate is actually able to execute a specific task, for example, whether it has access to the right infrastructure or connector. When Delegate tags are explicitly configured, this check matters less, but for configurations without tags, it is the primary mechanism for routing work correctly.
As part of ongoing work to support capability checking for a newer Delegate version, an optimization was introduced that skipped running a capability check whenever its underlying validation criteria came back empty, the assumption being that an empty criteria value meant there was nothing meaningful to validate, so the Delegate could be treated as already capable.
In practice, several existing capability checks had previously harmless defect, which under certain conditions would produce an empty criteria value despite still having a meaningful check that should run. Skipping checks in that state caused Delegates to be automatically treated as capable regardless of whether they actually were and most significantly, for legacy CI VM pipelines, where this logic determines whether a given Delegate is managing the correct virtual machine pool. With the check skipped, Delegates were marked capable of managing any VM pool, which caused CI tasks to be routed to Delegates that could not actually run them.
Because the underlying defects were pre-existing but previously harmless, the issue only surfaced once the new optimization began skipping the checks that had been quietly masking them, and it took several hours to appear widely because of a caching layer with a 6-hour refresh interval.
4. Resolution
Identified that CI VM task routing was being driven by faulty capability-check results.
Rolled back the backend deployment containing the optimization (the first rollback attempt did not fully take effect; a retry succeeded).
Provided customers with a configuration-based workaround (explicitly setting Delegate tags) to reduce impact while the rollback was in progress.
Reverted the capability-check optimization so checks with empty criteria run normally again rather than being skipped.
Added metrics and logging to help identify the specific capability checks that were producing empty criteria incorrectly, so they can be fixed at the source.
5. Preventive Actions
Fix the underlying capability checks that were producing empty criteria incorrectly (several affecting CI VM pipelines, with a few additional minor cases identified).
Safely re-introduce the capability-check optimization once those checks are fixed, since empty criteria should not normally occur once the underlying defects are resolved.
Add automated test coverage that exercises this CI VM Delegate-routing use case, since this was a regression that automated tests did not catch.
Add alerting specific to CI VM infrastructure routing, since no automated alert fired for this incident and it was first identified by customers.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.
Incident updated: We are currently seeing delegate selection issue for VM infra executions on Harness’s status page, changed 2026-11-10 23:03 UTC (change 5 of 5).
Title
Incident updated: We are currently seeing delegate selection issue for VM infra executions