ChangesTeleportDegraded Connectivity
Teleport: Major incident: Degraded Connectivity
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Teleport removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Degraded Connectivity
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-02-20 20:14 UTC
Before: not present
After: 2025-02-21 02:14 UTC
Before: not present
After: Cloud Service, Cloud Signups
Before: not present
After: We are currently investigating connectivity degradation on our infrastructure.
Before: not present
After: We've identified the issue and have confirmed that connectivity disruption is only affecting a subset of tenants. Additionally, new signups are currently unavailable until the issue is resolved.
Before: not present
After: Services have been restored in all regions.
Before: not present
After: This incident has been resolved.
Before: not present
After: Teleport Cloud customers experienced a service disruption intermittently impacting connectivity for about 40 minutes. The root cause of the incident was a bug in the Envoy Gateway ingress service that resulted in excessive memory utilization and ultimately restarts. The incident was triggered by the restart of all tenant services due to an undocumented background EKS storage migration that occurred during a scheduled maintenance period to upgrade the Kubernetes control plane.
In response to the incident, the Teleport team shut down ingress controllers which prevented routing updates from triggering ingress service OOM events. This reduced customer impact by stabilizing ingress connectivity. Once the EKS storage migration and tenant service restarts were completed, the team rolled back the Envoy gateway version which stabilized the platform.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Thursday, February 20, 2025
1 incident change
- MajorMajor incident
Degraded Connectivitythis change