ChangesTeleportSome clusters failing to restart following Tenant Upgrade
Teleport: Major incident: Some clusters failing to restart following Tenant Upgrade
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Teleport removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Some clusters failing to restart following Tenant Upgrade
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-05-04 14:20 UTC
Before: not present
After: 2023-05-04 16:21 UTC
Before: not present
After: Cloud Service
Before: not present
After: We've identified an issue where some customer Teleport clusters are failing to restart following Tenant Upgrade.
We're working on restoring services for affected customers.
Before: not present
After: We will be rolling back all customers who were upgraded yesterday due to an issue with access requests. Customers currently running 12.3.1 will be rolled back to 12.2.4. As a result, all active sessions to Teleport Cloud will be terminated. Teleport agents will automatically reconnect to their respective clusters. Connectivity may be disrupted for 2-10 minutes depending on the number of connected resources in the cluster.
Before: not present
After: Roll backs of customer Teleport clusters have been completed.
Before: not present
After: Following a routine upgrade of customer Teleport clusters to v12.3.1 a few tenants' Auth servers were failing healthy start up. It was determined that a change to the Okta access request code was leading to a panic. The affected tenants were rolled back. To prevent any other tenants from impact \(upon creation of an access request\) the balance of tenants were also downgraded to the previous version \(v12.2.4\).
Three tenants were affected by the inability to start their Auth service. Their service was disrupted for up to 2.5 hours. The remaining tenants had their services disrupted during remediation leading to an unplanned period of up to 10 minutes of service unavailability.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Thursday, May 4, 2023
1 incident change
- MajorMajor incident
Some clusters failing to restart following Tenant Upgradethis change