# Twingate: Critical incident: Inbound requests have heavily downgraded availability

**Summary:** _summary pending_
- Kind: Major incident · Label: critical (vendor)
- Changed: 2021-12-13 17:08 UTC · Detected: 2026-10-19 18:53 UTC
- [Source: https://stspg.io/d74tkpjx6tc7](https://stspg.io/d74tkpjx6tc7)
- [Company: Twingate](https://rewindex.com/company/twingate.md)
- Versions: before (none: first archived) → after incident\_version:758/8pbqr06npv5j@59ffb76d9917
- Change 1 of 1 for this incident

Incident page (full timeline): [Inbound requests have heavily downgraded availability](https://rewindex.com/company/twingate/incidents/8pbqr06npv5j.md)
## What changed

**1. Added: Title**

```diff
+ Inbound requests have heavily downgraded availability
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2021-12-13 17:08 UTC
```

**5. Added: Resolved time**

```diff
+ 2021-12-14 17:42 UTC
```

**6. Added: Affected components**

```diff
+ Connector Heartbeat, Twingate Controller
```

**7. Added: Update** (investigating) · 2021-12-13 17:08 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (investigating) · 2021-12-13 17:41 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (investigating) · 2021-12-13 18:07 UTC

```diff
+ We are continuing to investigate this issue. We will be posting regular updates pertaining to this incident.
```

**10. Added: Update** (investigating) · 2021-12-13 18:38 UTC

```diff
+ We are continuing to investigate this issue. We will be posting regular updates pertaining to this incident.
```

**11. Added: Update** (investigating) · 2021-12-13 19:08 UTC

```diff
+ We are continuing to investigate this issue. We have narrowed the source of the problem to the public-facing frontend servers that handle requests inbound to our service. As a result, this is broadly affecting our public API, the private API calls used by Clients and Connectors, and our web interface, resulting in heavily downgraded response availability across our service. We are still trying to identify the root cause at this time and will continue to post regular updates.
```

**12. Added: Update** (investigating) · 2021-12-13 19:12 UTC

```diff
+ Inbound requests are now being accepted and the service is operational again. We have verified that all operational tests are succeeding. We are continuing to investigate to determine the root cause of this incident.
```

**13. Added: Update** (monitoring) · 2021-12-13 19:21 UTC

```diff
+ We are continuing to monitor the system, and we are still investigating the root cause of the outage. We will continue to post additional updates regularly.
```

**14. Added: Update** (monitoring) · 2021-12-13 20:39 UTC

```diff
+ We are continuing to monitor the system, and it remains stable and available. We are investigating the root cause of the outage. We will continue to post additional updates regularly.
```

**15. Added: Update** (monitoring) · 2021-12-13 22:12 UTC

```diff
+ We are continuing to monitor the system, and it remains stable and available. We are investigating the root cause of the outage. We will post our next update at 17:00 PST / 01:00 UTC.
```

**16. Added: Update** (monitoring) · 2021-12-14 01:07 UTC

```diff
+ We are continuing to monitor the system, and it remains stable and available. We are investigating the root cause of the outage. We will post our next update at 21:00 PST / 05:00 UTC.
```

**17. Added: Update** (monitoring) · 2021-12-14 05:21 UTC

```diff
+ We are continuing to monitor the system, and it remains stable and available. We are investigating the root cause of the outage. We will post our next update at 09:00 PST / 17:00 UTC.
```

**18. Added: Update** (resolved) · 2021-12-14 17:42 UTC

```diff
+ We are continuing to monitor the system, and it remains stable and available. We are closing out this incident, and we will continue to post updates and follow up with a post mortem here.
```

**19. Added: Update** (postmortem) · 2021-12-21 04:35 UTC

```diff
+ **Components impacted** * Controller **Summary** At approximately 17:06 UTC on December 13th, we observed an increase in latency between our API layer and our backend database system. Within a few minutes, this spike in latency developed into an outage that resulted in 90-95% of requests returning one of two responses to the requestor: either a 500 \(Internal Server Error\) or a 502 \(Bad Gateway Error\) error depending on where the error in our system occurred. These error conditions were caused by timeouts occurring between our API layer and the database and persisted until approximately 19:08 UTC. During the outage, both our private and public APIs were affected. These APIs are used to drive most of the functionality that end users and administrators experience in Twingate. Specifically, this means that customers’ admin consoles were not accessible, the public API was not responsive to requests, Clients and Connectors were unable to initiate authentication, and existing connections were eventually dropped without the ability to re-authenticate. **Root cause** The root cause of the issue was due to temporary loss of connectivity and increased network latency in our cloud service provider between our API layer and backend database. The consequence of this increased latency was that even though our API layer was available to respond to requests, requests were taking significantly more time, leading to connection saturation of our API layer and the resulting rejection of additional requests, manifested as 500 or 502 errors to the requestor. **Corrective actions** In order to mitigate the risk of this root cause impacting our service in the future, we have initiated a number of improvements to isolate the impact of backend service disruptions from end user connectivity. These projects include decoupling our backend database from the Controller, scaling cross-regional database replicas for additional resiliency, and implementing changes to user connection behaviors to maintain connectivity in cases when backend services are unreachable.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2021-12-13 17:08 UTC | Critical incident: Inbound requests have heavily downgraded availability (this change) | critical |

All Twingate changes: [timeline](https://rewindex.com/company/twingate/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i65b6198d5dd283 · Markdown: https://rewindex.com/change/i65b6198d5dd283.md · JSON: https://rewindex.com/api/v1/changes/i65b6198d5dd283 · Generated 2026-10-19T21:44:34.166Z