ChangesTwingateController downtime
Twingate: Critical incident: Controller downtime
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Twingate removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Controller downtime
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2021-07-29 15:15 UTC
Before: not present
After: 2021-07-29 17:00 UTC
Before: not present
After: Connector Heartbeat, Twingate Controller
Before: not present
After: We've identified the issue, which is being caused by excessive memory usage on our infrastructure.
Before: not present
After: We're resolved the immediate issue by adding additional processing capacity and increasing memory limits on our controller infrastructructure.
Before: not present
After: The system is now confirmed as fully operational.
We are working on an incident report and taking steps to ensure that this issue will not happen in the future.
Before: not present
After: This incident has been resolved.
Before: not present
After: **Components impacted**
Controller
**Summary**
The controller was unavailable to service new authentication requests from approximately 15:17 to 15:19 UTC. The result was that during this period of time, new connection requests were rejected. Existing connections were not impacted. At the end of the outage period, normal service resumed with no remediation required.
**Root cause**
Leading up to the start of the outage period, automated monitoring alerted us to spikes in memory usage. At approximately 15:16 UTC we introduced a change to our cluster that was intended to increase memory availability. At approximately 15:17 UTC as this change was rolled out, it had the unintended consequence that resulted in a decrease in service availability, with the resulting rejection of most requests.
**Corrective actions**
At 15:18 UTC, seeing the decrease in service availability, we reverted the change and simultaneously made additional hardware available to the cluster. Normal service resumed approximately 45 seconds later as the change propagated.
Looking ahead, we plan to:
1. Investigate introducing decoupling between inbound requests and our backend as the likely cause of the memory spikes that triggered the change that caused the outage.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Thursday, July 29, 2021
1 incident change
- CriticalCritical incident
Controller downtimethis change