# Teleport: Major incident: Some clusters failing to restart following Tenant Upgrade

**Summary:** _summary pending_
- Kind: Major incident · Label: major (vendor)
- Changed: 2023-05-04 14:20 UTC · Detected: 2026-10-19 18:53 UTC
- [Source: https://stspg.io/zmnqtgzfzpr4](https://stspg.io/zmnqtgzfzpr4)
- [Company: Teleport](https://rewindex.com/company/teleport.md)
- Versions: before (none: first archived) → after incident\_version:757/hsgy5z6gzyvy@bc042e67135f
- Change 1 of 1 for this incident

Incident page (full timeline): [Some clusters failing to restart following Tenant Upgrade](https://rewindex.com/company/teleport/incidents/hsgy5z6gzyvy.md)
## What changed

**1. Added: Title**

```diff
+ Some clusters failing to restart following Tenant Upgrade
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2023-05-04 14:20 UTC
```

**5. Added: Resolved time**

```diff
+ 2023-05-04 16:21 UTC
```

**6. Added: Affected components**

```diff
+ Cloud Service
```

**7. Added: Update** (identified) · 2023-05-04 14:20 UTC

```diff
+ We've identified an issue where some customer Teleport clusters are failing to restart following Tenant Upgrade. We're working on restoring services for affected customers.
```

**8. Added: Update** (identified) · 2023-05-04 15:00 UTC

```diff
+ We will be rolling back all customers who were upgraded yesterday due to an issue with access requests. Customers currently running 12.3.1 will be rolled back to 12.2.4. As a result, all active sessions to Teleport Cloud will be terminated. Teleport agents will automatically reconnect to their respective clusters. Connectivity may be disrupted for 2-10 minutes depending on the number of connected resources in the cluster.
```

**9. Added: Update** (resolved) · 2023-05-04 16:21 UTC

```diff
+ Roll backs of customer Teleport clusters have been completed.
```

**10. Added: Update** (postmortem) · 2023-05-05 15:17 UTC

```diff
+ Following a routine upgrade of customer Teleport clusters to v12.3.1 a few tenants' Auth servers were failing healthy start up. It was determined that a change to the Okta access request code was leading to a panic. The affected tenants were rolled back. To prevent any other tenants from impact \(upon creation of an access request\) the balance of tenants were also downgraded to the previous version \(v12.2.4\). Three tenants were affected by the inability to start their Auth service. Their service was disrupted for up to 2.5 hours. The remaining tenants had their services disrupted during remediation leading to an unplanned period of up to 10 minutes of service unavailability.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2023-05-04 14:20 UTC | Major incident: Some clusters failing to restart following Tenant Upgrade (this change) | major |

All Teleport changes: [timeline](https://rewindex.com/company/teleport/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i5b465abee6c2e9 · Markdown: https://rewindex.com/change/i5b465abee6c2e9.md · JSON: https://rewindex.com/api/v1/changes/i5b465abee6c2e9 · Generated 2026-10-19T21:57:36.698Z