ChangesTwingateService provider networking outage
Twingate: Critical incident: Service provider networking outage
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
Part of an outage event
Google Cloud outage, Nov 16, 2021 · 24 vendors, 25 incidents
Twingate’s incident blames Google Cloud as the cause. (hop 1, posted +24 min vs the first vendor naming Google Cloud)
“…onnect their Connectors in order to restore access to protected resources. **Root cause** Google Cloud Platform \(GCP\) deployed a configuration change in their infrastructure that caused all requests…”
See the cascade chart and every vendor →
Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.
What changed
Struck in red is what Twingate removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Service provider networking outage
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2021-11-16 17:53 UTC
Before: not present
After: 2021-11-16 22:41 UTC
Before: not present
After: Americas Relays, Connector Heartbeat, Twingate Controller, Twingate Relays
Before: not present
After: We are investigating an outage report with regards to Twingate. At this time we suspect it is an issue affecting broader Internet services and is not isolated to Twingate. We will continue to post regular updates as we learn more.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: The Twingate admin console is now accessible and the Twingate Controller is operational.
Customers may need to restart Connectors to restore connectivity to resources due to the nature of the networking outage.
The originating cause appears to be related to an outage in Google Cloud Platform's Networking service.
Google Cloud Platform has opened an incident: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh
Before: not present
After: We are continuing to monitor the status of the service.
Customers may need to restart Connectors to restore connectivity to resources due to the nature of the networking outage.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: We have verified that all of our infrastructure is fully operational at this time and will continue to monitor for any changes.
Until our service provider (Google Cloud Platform) has closed their incident, we will leave this incident open in Monitoring status and provide regular updates as we receive them.
Customers should verify that all of their Connectors are up and running if any Resources are inaccessible at this time.
Before: not present
After: Google Cloud Platform has marked their Cloud Networking issue as resolved and has posted a status update: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh
We are continuing to monitor our infrastructure and will mark this incident as resolved when we are confident that everything has returned to normal.
Before: not present
After: We are marking this issue as resolved as our monitoring shows that our infrastructure is operating normally and Google Cloud Platform has resolved the incident on their network.
We will be following up with a post-mortem shortly.
Before: not present
After: **Components impacted**
Controller
Relays
**Summary**
Twingate services were unavailable to service requests from approximately 17:48 to 20:09 UTC on November 16th. The result was that during this period of time, access to Twingate and protected resources was limited, existing connections were dropped, and new connections were refused. At the end of the period, normal service resumed. Remediation required that customers reconnect their Connectors in order to restore access to protected resources.
**Root cause**
Google Cloud Platform \(GCP\) deployed a configuration change in their infrastructure that caused all requests to return 404 errors \([GCP incident description](https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh)\). Because Twingate relies on GCP infrastructure, access to the Twingate network and protected resources was impacted. GCP confirms that the incident was resolved as of 19:28 UTC. As GCP began to restore their service, impacted Twingate services automatically came back online. Currently, Twingate Clients and Connectors view 404 errors as unrecoverable states and thus did not automatically reconnect. Consequently, customers were required to restart their Connectors and the Windows service on the Windows Client to restore access.
**Corrective actions**
Automated monitoring alerted Twingate to the outage and our DevOps and on-call engineering teams started tracking the issue. Manual testing confirmed the outage, and additional investigation showed that other GCP customers were impacted. While traffic was being restored, systems indicated that Connectors did not automatically recover. For customers using our Managed Connectors, these were restarted at 20:50 UTC. We began notifying customers about the need to restart Connectors at approximately 19:00 UTC, and all customers were notified by 02:02 UTC on November 17th.
Looking ahead, we plan to:
* Prioritize Client and Connector reconnection behavior and extend it to include all non-recoverable errors
* Introduce functionality to notify customers of Connector downtime via email notifications
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Tuesday, November 16, 2021
1 incident change
- CriticalCritical incident
Service provider networking outagethis change