Teleport: Degraded Connectivity
Affected components: Cloud SignupsCloud Service
Timeline of updates
- investigating
We are currently investigating connectivity degradation on our infrastructure.
- identified
We've identified the issue and have confirmed that connectivity disruption is only affecting a subset of tenants. Additionally, new signups are currently unavailable until the issue is resolved.
- monitoring
Services have been restored in all regions.
- resolved
This incident has been resolved.
- postmortem
Teleport Cloud customers experienced a service disruption intermittently impacting connectivity for about 40 minutes. The root cause of the incident was a bug in the Envoy Gateway ingress service that resulted in excessive memory utilization and ultimately restarts. The incident was triggered by the restart of all tenant services due to an undocumented background EKS storage migration that occurred during a scheduled maintenance period to upgrade the Kubernetes control plane. In response to the incident, the Teleport team shut down ingress controllers which prevented routing updates from triggering ingress service OOM events. This reduced customer impact by stabilizing ingress connectivity. Once the EKS storage migration and tenant service restarts were completed, the team rolled back the Envoy gateway version which stabilized the platform.
Changes Rewindex caught on this incident
- Feb 20, 2025Major incident: Degraded Connectivity
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).