rewindex

ChangesHarnessIntermittent failures in Policy step evaluations on Prod 7

Harness: Incident updated: Intermittent failures in Policy step evaluations on Prod 7

Incident updateNot classified yetChanged stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what Harness removed; blue is what they added. Muted text is unchanged context.

ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Summary Policy evaluation for a pipeline step intermittently failed for one customer with an "unsupported principal type for internal token" error once an enhanced tenant-isolation safeguard was turned on. Root cause Our internal services authenticate to each other using signed internal tokens. The service that evaluates policy only accepts tokens whose principal is marked as one of a small set of recognized types; a token with a missing or unrecognized principal type is rejected. The affected pipeline step was missing a safeguard that other callers rely on to guarantee their internal token always carries a recognized principal type. As a result, under certain conditions it could present a token with an unsupported principal type to the policy service, causing the request to be rejected. ‌ Mitigation We mitigated the immediate impact by disabling the safeguard. Remediation We are applying the fix at the shared client layer used by every caller of the policy evaluation service, including the affected pipeline step, OPA helper utilities, secrets, and connectors, so the fix protects all current and future callers consistently. Tokens with a recognized principal type continue to be forwarded as-is. Tokens with a missing or unsupported principal type will instead be re-signed as the calling service. Next steps We are rolling out the fix gradually, starting in shadow mode, with additional logging and observability added across all call paths so we can confirm the fix covers every use case, including lower-traffic ones, before it is enabled broadly.

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdayTuesday, November 10Tue, Nov 10

1 incident change

Cite this

Incident updated: Intermittent failures in Policy step evaluations on Prod 7 on Harness’s status page, changed 2026-11-10 22:23 UTC (change 1 of 1).

Title
Incident updated: Intermittent failures in Policy step evaluations on Prod 7
Vendor
Harness
Date
URL
https://rewindex.com/change/r4b5a492b96ce17

All Harness changesHarness profileMarkdownJSON