# Twingate: Twingate Service Issues

- Impact (vendor’s label): minor
- Status: postmortem
- Started: 2025-03-21 19:57 UTC · Resolved: 2025-03-21 21:13 UTC · Duration: 1.3 h
- Updates: 4 · Affected components: Authorization, Control Plane - Authorization, Admin Console, Management - Admin Console, DNS Filtering Dashboards, Management - DNS Filtering Dashboards, 3rd Party Integrations, Management - 3rd Party Integrations, Management - Public API, Public API, Management - Realtime Updates, Realtime Updates, Identity Providers Sync, Management - Identity Providers Sync, Control Plane - MFA, MFA, Client Log Upload, Management - Client Log Upload, Authentication - Social, Control Plane - Authentication - Social, Management - Network Dashboards, Network Dashboards, Authentication - Enterprise, Control Plane - Authentication - Enterprise, Connector Heartbeat, Control Plane - Connector Heartbeat
- [Vendor status page: https://stspg.io/ktzm8qc6j363](https://stspg.io/ktzm8qc6j363)
- [Company: Twingate](https://rewindex.com/company/twingate.md)

## Timeline of updates

- **investigating** 2025-03-21 20:10 UTC: We are currently investigating this issue.
- **monitoring** 2025-03-21 20:41 UTC: A fix has been implemented and we are monitoring the results.
- **resolved** 2025-03-21 21:13 UTC: This incident has been resolved.
- **postmortem** 2025-03-28 15:14 UTC: \*\*Twingate Public RCA: March 21, 2025 Authentication/Authorization Incident\*\* \*\*Summary\*\* On March 21, 2025, between 21:10 and 21:41 UTC, a subset of authentication and authorization requests to Twingate services experienced elevated error rates. The issue was mitigated by 21:41 UTC and services returned to normal operation. \*\*What Happened\*\* Twingate runs an active-active architecture across multiple Google Kubernetes Engine \\(GKE\\) clusters, balanced by a Global Load Balancer \\(GLB\\). At the onset of the incident, we observed anomalies in one of our clusters, and to protect overall system health, we proactively scaled down deployments in that cluster. This action shifted traffic to other clusters in the topology, primarily one that typically handles a lighter load and had been scaled accordingly. The target cluster began autoscaling as expected, but the increased traffic caused elevated error rates, which triggered our retry mechanisms across services. While these retries helped many requests succeed, they also increased the overall system load. Simultaneously, the cluster underwent a cloud provider-initiated update operation that caused pod restarts and reduced capacity. To stabilize the system, we reintroduced capacity in the previously affected cluster, rebalancing the traffic across regions. Once this occurred, error rates subsided and retries diminished. \*\*Root Cause\*\* Anomalous behavior from our cloud provider, including unexpected request timeouts at the load balancer level and instability during a cluster update, led to a cascade of retry traffic that temporarily overwhelmed parts of the system. We are actively investigating both the unexpected timeout configuration and the behavior of the cluster during the update with our cloud provider. \*\*Corrective actions\*\* Short-Term: \* Continue collaboration with our cloud provider to understand the root cause of the unexpected timeouts and cluster update impact. \* COMPLETED: Keep deployments on all regions with same number of replicas in HPA configuration \* COMPLETED: Increase max node counts on cluster node auto-scalers to give us more room to scale up. \* COMPLETED: Configure HPA to scale up faster


---
Canonical: https://rewindex.com/company/twingate/incidents/mhs89xs5rhb8 · Markdown: https://rewindex.com/company/twingate/incidents/mhs89xs5rhb8.md · JSON: https://rewindex.com/api/v1/companies/twingate/incidents/mhs89xs5rhb8 · Generated 2026-10-19T21:09:55.591Z