# Harness: Incident updated: We are currently seeing delegate selection issue for VM infra executions

> Incident updated: We are currently seeing delegate selection issue for VM infra executions. Detected 2026-11-10 by Rewindex: what changed on Harness’s status page, with the before/after diff.

Canonical: https://rewindex.com/change/rbb71aa1dc0c2df
Last updated: 2026-11-10T23:03:53Z

**Summary:** _summary pending_
- Kind: Incident update · Label: not classified yet
- Changed: 2026-11-10 23:03 UTC · Detected: 2026-11-10 23:03 UTC
- [Source: https://status.harness.io/incidents/jmfpjjljc0n3](https://status.harness.io/incidents/jmfpjjljc0n3)
- [Company: Harness](https://rewindex.com/company/harness.md)
- Versions: before incident\_version:695/jmfpjjljc0n3@12a0657a3207 → after incident\_version:695/jmfpjjljc0n3@4e9d80832685
- Change 5 of 5 for this incident

Incident page (full timeline): [We are currently seeing delegate selection issue for VM infra executions](https://rewindex.com/company/harness/incidents/jmfpjjljc0n3.md)
## What changed

**1. Changed: Status**

```diff
- resolved
+ postmortem
```

**2. Added: Update** (postmortem) · 2026-11-10 22:55 UTC

```diff
+ 1. Summary On October 27, 2026, starting around 6:35 AM, customers running CI pipelines on self-managed VM infrastructure (a legacy Delegate-based VM runner configuration) in Prod1 began experiencing pipeline failures caused by tasks being routed to Delegates that could not actually execute them. The issue affected a small number of customers using this specific, less common configuration (VM-based CI without Delegate tags configured). It was introduced by a backend deployment and was mitigated by rolling that deployment back; a permanent fix followed. 2. Impact CI pipelines using VM-based execution without Delegate tags configured began intermittently failing to run on affected Delegates. Other pipeline types and configurations were not affected. Impact was limited to a small number of customers in Prod1 using this specific configuration. The partial outage lasted approximately 8 hours before being resolved via rollback. Affected customers were given a configuration-based workaround (setting Delegate tags) during the incident. 3. Root Cause Our Delegate platform uses a feature called Capability Checks to determine whether a given Delegate is actually able to execute a specific task,  for example, whether it has access to the right infrastructure or connector. When Delegate tags are explicitly configured, this check matters less, but for configurations without tags, it is the primary mechanism for routing work correctly. ‌ As part of ongoing work to support capability checking for a newer Delegate version, an optimization was introduced that skipped running a capability check whenever its underlying validation criteria came back empty, the assumption being that an empty criteria value meant there was nothing meaningful to validate, so the Delegate could be treated as already capable. ‌ In practice, several existing capability checks had previously harmless defect, which under certain conditions would produce an empty criteria value despite still having a meaningful check that should run. Skipping checks in that state caused Delegates to be automatically treated as capable regardless of whether they actually were and most significantly, for legacy CI VM pipelines, where this logic determines whether a given Delegate is managing the correct virtual machine pool. With the check skipped, Delegates were marked capable of managing any VM pool, which caused CI tasks to be routed to Delegates that could not actually run them. ‌ Because the underlying defects were pre-existing but previously harmless, the issue only surfaced once the new optimization began skipping the checks that had been quietly masking them, and it took several hours to appear widely because of a caching layer with a 6-hour refresh interval. 4. Resolution Identified that CI VM task routing was being driven by faulty capability-check results. Rolled back the backend deployment containing the optimization (the first rollback attempt did not fully take effect; a retry succeeded). Provided customers with a configuration-based workaround (explicitly setting Delegate tags) to reduce impact while the rollback was in progress. Reverted the capability-check optimization so checks with empty criteria run normally again rather than being skipped. Added metrics and logging to help identify the specific capability checks that were producing empty criteria incorrectly, so they can be fixed at the source. 5. Preventive Actions Fix the underlying capability checks that were producing empty criteria incorrectly (several affecting CI VM pipelines, with a few additional minor cases identified). Safely re-introduce the capability-check optimization once those checks are fixed, since empty criteria should not normally occur once the underlying defects are resolved. Add automated test coverage that exercises this CI VM Delegate-routing use case, since this was a regression that automated tests did not catch. Add alerting specific to CI VM infrastructure routing, since no automated alert fired for this incident and it was first identified by customers.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-10-27 20:04 UTC | [Incident updated: We are currently seeing delegate selection issue for VM infra executions](https://rewindex.com/change/r26991fad1f5a69.md) | Normal progress |
| 2026-10-27 21:24 UTC | [Incident updated: We are currently seeing delegate selection issue for VM infra executions](https://rewindex.com/change/r479d36bc0cadad.md) | Normal progress |
| 2026-10-27 22:24 UTC | [Incident updated: We are currently seeing delegate selection issue for VM infra executions](https://rewindex.com/change/r33ec4c525a5be4.md) | Normal progress |
| 2026-10-27 22:34 UTC | [Incident updated: We are currently seeing delegate selection issue for VM infra executions](https://rewindex.com/change/r4b40a06ce0c7fe.md) | Normal progress |
| 2026-11-10 23:03 UTC | Incident updated: We are currently seeing delegate selection issue for VM infra executions (this change) |  |

Previous: [change](https://rewindex.com/change/r4b40a06ce0c7fe.md) · All Harness changes: [timeline](https://rewindex.com/company/harness/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._

## Cite this

Incident updated: We are currently seeing delegate selection issue for VM infra executions on Harness’s status page, changed 2026-11-10 23:03 UTC (change 5 of 5).

- Canonical URL: https://rewindex.com/change/rbb71aa1dc0c2df
- Citation: Harness, “Incident updated: We are currently seeing delegate selection issue for VM infra executions” (2026-11-10 23:03 UTC). Archived by Rewindex: https://rewindex.com/change/rbb71aa1dc0c2df (CC BY 4.0).
- Markdown link: `[Harness: Incident updated: We are currently seeing delegate selection issue for VM infra executions (2026-11-10)](https://rewindex.com/change/rbb71aa1dc0c2df)`


---
Canonical: https://rewindex.com/change/rbb71aa1dc0c2df · Markdown: https://rewindex.com/change/rbb71aa1dc0c2df.md · JSON: https://rewindex.com/api/v1/changes/rbb71aa1dc0c2df · Data licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) · Generated 2026-11-11T04:20:44.928Z