# Harness: Critical incident: MTLS Outage with Delegates

**Summary:** _summary pending_
- Kind: Major incident · Label: critical (vendor)
- Changed: 2026-09-14 19:00 UTC · Detected: 2026-10-19 17:33 UTC
- [Source: https://stspg.io/6mmv855y29p7](https://stspg.io/6mmv855y29p7)
- [Company: Harness](https://rewindex.com/company/harness.md)
- Versions: before (none: first archived) → after incident\_version:695/krb186nq1nwq@4c1cbee51971
- Change 1 of 1 for this incident

Incident page (full timeline): [MTLS Outage with Delegates](https://rewindex.com/company/harness/incidents/krb186nq1nwq.md)
## What changed

**1. Added: Title**

```diff
+ MTLS Outage with Delegates
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2026-09-14 19:00 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-09-15 03:28 UTC
```

**6. Added: Affected components**

```diff
+ Continuous Delivery - Next Generation (CDNG), Prod 1 - Continuous Delivery - Next Generation (CDNG), Prod 2 - Continuous Delivery - Next Generation (CDNG), Prod 3 - Continuous Delivery - Next Generation (CDNG)
```

**7. Added: Update** (investigating) · 2026-09-14 19:00 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (identified) · 2026-09-15 00:42 UTC

```diff
+ The issue has been identified and a fix is being implemented.
```

**9. Added: Update** (identified) · 2026-09-15 00:45 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**10. Added: Update** (monitoring) · 2026-09-15 01:46 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**11. Added: Update** (resolved) · 2026-09-15 03:28 UTC

```diff
+ This incident has been resolved.
```

**12. Added: Update** (postmortem) · 2026-09-21 18:26 UTC

```diff
+ # **Summary** To improve the security posture, On September 14, 2026, Harness performed a scheduled rotation of the TLS certificate used to secure Delegate connections for accounts using mutual TLS \(mTLS\),The replacement   certificate passed Harness's existing pre-deployment validation and smoke testing. However, the rotation also   introduced a change in certificate authority, which required additional certificate-chain and gateway   configuration this prevented affected Delegates from connecting to Harness.   Harness has identified the contributing factors and is implementing the corrective actions described below to prevent a recurrence.   # **Impact**    Delegates configured to use mutual TLS in Prod environment were unable to connect to Harness for   until Harness applied a temporary mitigation \(disabling mTLS\) that restored   connectivity while the underlying issue was corrected. This affected the ability to run pipelines and other operations dependent on the affected Delegates. Full,   permanent connectivity — with mTLS restored to its normal, fully secure configuration — was confirmed for   both environments by 7:20 PM PST.   # **Root Cause** ‌    As part of routine certificate maintenance, Harness rotated the certificate used by the gateway component that   secures mutual TLS connections for Delegates in prod environments.  The rotation also introduced a   change in certificate authority, which required additional certificate-chain and gateway configuration that was   not fully covered by the existing validation process. As a result, the gateway was unable to load the new   certificate correctly, and Delegates relying on mutual TLS could not establish a secure connection.   # **Remediation**    Once the certificate rotation was confirmed as the source of the issue, Harness engineering worked continuously   to correct the certificate and validate the fix with Citi before fully restoring the secure configuration.   Corrective and Preventive Actions During remediation, Harness also identified and corrected a secondary issue with the replacement certificate's   chain, which required an additional deployment before connectivity was fully and permanently restored.   # **Next Steps**    Harness is taking the following steps to prevent a recurrence of this issue: 1. Extend pre-deployment validation and smoke testing to explicitly cover certificate authority changes and full     certificate-chain verification, closing the specific gap that allowed this configuration through. 2. Enhance automation  for managing mTLS-related configuration changes, 3. Improve monitoring and alerting for Delegate connectivity, including account-level visibility, so similar issues     are detected automatically
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-09-14 19:00 UTC | Critical incident: MTLS Outage with Delegates (this change) | critical |

All Harness changes: [timeline](https://rewindex.com/company/harness/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i2cadd99002ec3f · Markdown: https://rewindex.com/change/i2cadd99002ec3f.md · JSON: https://rewindex.com/api/v1/changes/i2cadd99002ec3f · Generated 2026-10-19T23:13:08.889Z