Twingate: Twingate Service Incident - Aug 19, 2023
Affected components: AuthorizationController - Connection AuthorizationAdmin ConsoleAdmin Console - Admin Web InterfaceAdmin APIPublic APIAuthentication - EnterpriseController - User AuthenticationConnector HeartbeatController - Controller
Timeline of updates
- investigating
We are seeing issues with Twingate service and investigating.
- investigating
We are continuing to investigate this issue.
- resolved
This incident has been resolved. We'll publish RCA as soon as we can.
- postmortem
**Summary** On August 19 at 7:51 AM UTC, Twingate received alerts of issues with the login services. Within a few minutes, the Twingate engineering team began investigating. The team quickly identified that our backend was seeing excessive timeouts from a 3rd-party API, preventing it from being able to process other requests such as authentication. After some initial fixes were unsuccessful, Twingate contacted the 3rd party and also disabled support for real-time updates that make use of these specific 3rd-party API calls. As a result, the issues started resolving at 8:10 AM UTC. Most of the services recovered quickly and full resolution occurred at 8:15 AM UTC. The vendor later confirmed and fixed the issue, and Twingate re-enabled the real-time update feature shortly after on the same day, August 19. **Root cause** The Twingate backend was exhausted due to timeouts from a 3rd-party API. **Post-incident Analysis** Twingate had already separated out most services to their own deployments, allowing those services to function throughout the incident. Therefore, only some users that needed to authenticate or re-authenticate were affected; any user that had authenticated prior to the incident was not impacted. Analysis of logs post-incident showed that the incident started at 7:49 AM UTC and fully recovered at 8:15 AM UTC. **Corrective actions** Short Term: * Separate Authentication and real-time services to their own deployments - COMPLETED Medium / Long Term: * Reevaluate and optimize timeout values for various backend and 3rd party services * Simplify the internal Twingate process for enabling and disabling features
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).