# Teleport: Some clusters failing to restart following Tenant Upgrade

- Impact (vendor’s label): major
- Status: postmortem
- Started: 2023-05-04 14:20 UTC · Resolved: 2023-05-04 16:21 UTC · Duration: 2.0 h
- Updates: 4 · Affected components: Cloud Service
- [Vendor status page: https://stspg.io/zmnqtgzfzpr4](https://stspg.io/zmnqtgzfzpr4)
- [Company: Teleport](https://rewindex.com/company/teleport.md)

## Timeline of updates

- **identified** 2023-05-04 14:20 UTC: We've identified an issue where some customer Teleport clusters are failing to restart following Tenant Upgrade. We're working on restoring services for affected customers.
- **identified** 2023-05-04 15:00 UTC: We will be rolling back all customers who were upgraded yesterday due to an issue with access requests. Customers currently running 12.3.1 will be rolled back to 12.2.4. As a result, all active sessions to Teleport Cloud will be terminated. Teleport agents will automatically reconnect to their respective clusters. Connectivity may be disrupted for 2-10 minutes depending on the number of connected resources in the cluster.
- **resolved** 2023-05-04 16:21 UTC: Roll backs of customer Teleport clusters have been completed.
- **postmortem** 2023-05-05 15:17 UTC: Following a routine upgrade of customer Teleport clusters to v12.3.1 a few tenants' Auth servers were failing healthy start up. It was determined that a change to the Okta access request code was leading to a panic. The affected tenants were rolled back. To prevent any other tenants from impact \\(upon creation of an access request\\) the balance of tenants were also downgraded to the previous version \\(v12.2.4\\). Three tenants were affected by the inability to start their Auth service. Their service was disrupted for up to 2.5 hours. The remaining tenants had their services disrupted during remediation leading to an unplanned period of up to 10 minutes of service unavailability.

## Rewindex changes for this incident

- 2023-05-04 14:20 UTC: [Major incident: Some clusters failing to restart following Tenant Upgrade](https://rewindex.com/change/i5b465abee6c2e9.md)


---
Canonical: https://rewindex.com/company/teleport/incidents/hsgy5z6gzyvy · Markdown: https://rewindex.com/company/teleport/incidents/hsgy5z6gzyvy.md · JSON: https://rewindex.com/api/v1/companies/teleport/incidents/hsgy5z6gzyvy · Generated 2026-10-19T21:15:10.573Z