# Twingate: Critical incident: Twingate Service Down

**Summary:** _summary pending_
- Kind: Major incident · Label: critical (vendor)
- Changed: 2025-05-28 20:53 UTC · Detected: 2026-10-19 18:53 UTC
- [Source: https://stspg.io/clk4dmf2zptn](https://stspg.io/clk4dmf2zptn)
- [Company: Twingate](https://rewindex.com/company/twingate.md)
- Versions: before (none: first archived) → after incident\_version:758/d57tmhvs1c5x@6671041845c5
- Change 1 of 1 for this incident

Incident page (full timeline): [Twingate Service Down](https://rewindex.com/company/twingate/incidents/d57tmhvs1c5x.md)
## What changed

**1. Added: Title**

```diff
+ Twingate Service Down
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2025-05-28 20:53 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-05-28 21:32 UTC
```

**6. Added: Affected components**

```diff
+ 3rd Party Integrations, Admin Console, Authentication - Enterprise, Authentication - Social, Authorization, Client Log Upload, Connector Heartbeat, Control Plane - Authentication - Enterprise, Control Plane - Authentication - Social, Control Plane - Authorization, Control Plane - Connector Heartbeat, Control Plane - MFA, DNS Filtering Dashboards, Identity Providers Sync, Management - 3rd Party Integrations, Management - Admin Console, Management - Client Log Upload, Management - DNS Filtering Dashboards, Management - Identity Providers Sync, Management - Network Dashboards, Management - Public API, Management - Realtime Updates, MFA, Network Dashboards, Public API, Realtime Updates
```

**7. Added: Update** (identified) · 2025-05-28 20:53 UTC

```diff
+ The issue has been identified and a fix is being implemented.
```

**8. Added: Update** (monitoring) · 2025-05-28 21:16 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**9. Added: Update** (resolved) · 2025-05-28 21:32 UTC

```diff
+ This incident is resolved now. We'll provide a post-mortem as soon as we have details.
```

**10. Added: Update** (postmortem) · 2025-06-16 18:26 UTC

```diff
+ ### **Incident Duration** **UTC Time**: May 28, 2025, from **20:45 to 21:14** \(29 minutes\) ### **Components Impacted** **Control Plane Services** * Authentication \(Enterprise & Social\) * Multi-Factor Authentication \(MFA\) * Authorization * Connector Heartbeat **Management Plane Services** * Identity Provider Sync * Public API * Admin Console * Real-Time Updates * Client Log Upload * 3rd Party Integrations * Network Dashboards * DNS Filtering Dashboards ### **Summary** On May 28, 2025, Twingate experienced a service disruption that impacted multiple core services for approximately 29 minutes. The issue stemmed from a misconfiguration introduced during a rollout that updated routing rules in our global load balancer. A new routing rule was unintentionally prioritized, resulting in most API traffic being directed to a single backend. This overwhelmed the target service, leading to elevated error rates. Lower environments did not expose this issue due to their reduced traffic levels. The incident was promptly detected and the rollback was initiated quickly. However, recovery was delayed due to infrastructure dependencies that also relied on affected components. A manual intervention using local tooling ultimately restored service. ### **Root Cause** A routing misconfiguration caused a traffic imbalance that overwhelmed a backend service, leading to widespread API failures. ### **Resolution Timeline** * **20:45 UTC** – Incident begins. API traffic misrouted. * **20:48 UTC** – Incident detected and triage begins. * **20:52 UTC** – Rollback initiated. * **21:05 UTC** – Automated rollback process encounters delays. * **21:09 UTC** – Engineers execute manual fix. * **21:14 UTC** – Services fully recovered. ### **Corrective and Preventative Actions** ### **Short-Term \(Completed\)** * ✅ Routing rule corrected. * ✅ Alerts added to detect backend traffic anomalies in lower environments. * ✅ Improved resource scaling thresholds. ### **Mid-Term \(In Progress\)** * ⚙️ Decouple infrastructure tooling from service dependencies to ensure faster recovery options. ### **Long-Term \(Planned\)** * 🔄 Introduce backend sharding to isolate customer workloads and reduce incident impact. We sincerely apologize for the disruption and are taking actions to improve our systems' resilience. We appreciate your continued support and trust.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-05-28 20:53 UTC | Critical incident: Twingate Service Down (this change) | critical |

All Twingate changes: [timeline](https://rewindex.com/company/twingate/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i14b0c57a16a412 · Markdown: https://rewindex.com/change/i14b0c57a16a412.md · JSON: https://rewindex.com/api/v1/changes/i14b0c57a16a412 · Generated 2026-10-19T21:44:31.589Z