# Twingate: Critical incident: Service provider networking outage **Summary:** _summary pending_ ## Part of an outage event - [Google Cloud outage, Nov 16, 2021: 24 vendors, 25 incidents](https://rewindex.com/events/2021-11-16-google-cloud) - This incident blames Google Cloud as the cause. (hop 1, posted +24 min vs the first vendor naming Google Cloud) - Evidence: “…onnect their Connectors in order to restore access to protected resources. \*\*Root cause\*\* Google Cloud Platform \\(GCP\\) deployed a configuration change in their infrastructure that caused all requests…” ```mermaid flowchart LR root["Google Cloud
own incident not archived"]:::inferred h1["Within an hour
6 vendors, 4 blamed Google Cloud
Segment, Bugsnag, Twingate, Sigma Computing +2"]:::blames root --> h1 h4["1 to 4 hours later
2 vendors, 2 blamed Google Cloud
Smartsheet, Grafana Cloud"]:::blames root --> h4 me["Twingate (this incident)
posted +24 min"]:::me h1 --- me ``` _Detected automatically from incidents that started together and named the same provider; a link means one incident named another vendor, not a confirmed cause._ - Kind: Major incident · Label: critical (vendor) - Changed: 2021-11-16 17:53 UTC · Detected: 2026-10-19 18:53 UTC - [Source: https://stspg.io/mplr86mk5zzr](https://stspg.io/mplr86mk5zzr) - [Company: Twingate](https://rewindex.com/company/twingate.md) - Versions: before (none: first archived) → after incident\_version:758/zgknr5nmpk8h@2dba4dc59dc5 - Change 1 of 1 for this incident Incident page (full timeline): [Service provider networking outage](https://rewindex.com/company/twingate/incidents/zgknr5nmpk8h.md) ## What changed **1. Added: Title** ```diff + Service provider networking outage ``` **2. Added: Impact** ```diff + critical ``` **3. Added: Status** ```diff + postmortem ``` **4. Added: Start time** ```diff + 2021-11-16 17:53 UTC ``` **5. Added: Resolved time** ```diff + 2021-11-16 22:41 UTC ``` **6. Added: Affected components** ```diff + Americas Relays, Connector Heartbeat, Twingate Controller, Twingate Relays ``` **7. Added: Update** (investigating) · 2021-11-16 17:53 UTC ```diff + We are investigating an outage report with regards to Twingate. At this time we suspect it is an issue affecting broader Internet services and is not isolated to Twingate. We will continue to post regular updates as we learn more. ``` **8. Added: Update** (investigating) · 2021-11-16 18:06 UTC ```diff + We are continuing to investigate this issue. ``` **9. Added: Update** (investigating) · 2021-11-16 18:16 UTC ```diff + The Twingate admin console is now accessible and the Twingate Controller is operational. Customers may need to restart Connectors to restore connectivity to resources due to the nature of the networking outage. The originating cause appears to be related to an outage in Google Cloud Platform's Networking service. Google Cloud Platform has opened an incident: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh ``` **10. Added: Update** (monitoring) · 2021-11-16 18:29 UTC ```diff + We are continuing to monitor the status of the service. Customers may need to restart Connectors to restore connectivity to resources due to the nature of the networking outage. ``` **11. Added: Update** (monitoring) · 2021-11-16 18:29 UTC ```diff + We are continuing to monitor for any further issues. ``` **12. Added: Update** (monitoring) · 2021-11-16 19:08 UTC ```diff + We have verified that all of our infrastructure is fully operational at this time and will continue to monitor for any changes. Until our service provider (Google Cloud Platform) has closed their incident, we will leave this incident open in Monitoring status and provide regular updates as we receive them. Customers should verify that all of their Connectors are up and running if any Resources are inaccessible at this time. ``` **13. Added: Update** (monitoring) · 2021-11-16 20:16 UTC ```diff + Google Cloud Platform has marked their Cloud Networking issue as resolved and has posted a status update: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh We are continuing to monitor our infrastructure and will mark this incident as resolved when we are confident that everything has returned to normal. ``` **14. Added: Update** (resolved) · 2021-11-16 22:41 UTC ```diff + We are marking this issue as resolved as our monitoring shows that our infrastructure is operating normally and Google Cloud Platform has resolved the incident on their network. We will be following up with a post-mortem shortly. ``` **15. Added: Update** (postmortem) · 2021-11-19 23:53 UTC ```diff + **Components impacted** Controller Relays **Summary** Twingate services were unavailable to service requests from approximately 17:48 to 20:09 UTC on November 16th. The result was that during this period of time, access to Twingate and protected resources was limited, existing connections were dropped, and new connections were refused. At the end of the period, normal service resumed. Remediation required that customers reconnect their Connectors in order to restore access to protected resources. **Root cause** Google Cloud Platform \(GCP\) deployed a configuration change in their infrastructure that caused all requests to return 404 errors \([GCP incident description](https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh)\). Because Twingate relies on GCP infrastructure, access to the Twingate network and protected resources was impacted. GCP confirms that the incident was resolved as of 19:28 UTC. As GCP began to restore their service, impacted Twingate services automatically came back online. Currently, Twingate Clients and Connectors view 404 errors as unrecoverable states and thus did not automatically reconnect. Consequently, customers were required to restart their Connectors and the Windows service on the Windows Client to restore access. **Corrective actions** Automated monitoring alerted Twingate to the outage and our DevOps and on-call engineering teams started tracking the issue. Manual testing confirmed the outage, and additional investigation showed that other GCP customers were impacted. While traffic was being restored, systems indicated that Connectors did not automatically recover. For customers using our Managed Connectors, these were restarted at 20:50 UTC. We began notifying customers about the need to restart Connectors at approximately 19:00 UTC, and all customers were notified by 02:02 UTC on November 17th. Looking ahead, we plan to: * Prioritize Client and Connector reconnection behavior and extend it to include all non-recoverable errors * Introduce functionality to notify customers of Connector downtime via email notifications ``` ## History of this incident | When (UTC) | Change | Label | | --- | --- | --- | | 2021-11-16 17:53 UTC | Critical incident: Service provider networking outage (this change) | critical | All Twingate changes: [timeline](https://rewindex.com/company/twingate/changes.md) _Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._ --- Canonical: https://rewindex.com/change/id8aee8556760b4 · Markdown: https://rewindex.com/change/id8aee8556760b4.md · JSON: https://rewindex.com/api/v1/changes/id8aee8556760b4 · Generated 2026-10-19T21:43:15.687Z