ChangesTwingateAdmin console billing functionality is temporarily unavailable
Twingate: Major incident: Admin console billing functionality is temporarily unavailable
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Twingate removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Admin console billing functionality is temporarily unavailable
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2021-12-15 19:12 UTC
Before: not present
After: 2021-12-15 21:22 UTC
Before: not present
After: Admin Console
Before: not present
After: The admin console is currently experiencing an outage caused by our 3rd party billing system being unavailable. Our team is investigating the issue and working on a fix currently.
This issue is isolated to loading the admin console and does not affect any resource access or end user authentication.
Before: not present
After: The root cause of the issue has been identified and a fix is in progress. We will post another update shortly.
Before: not present
After: A fix has been deployed and the admin console is now available to all customers. We'll continue to monitor the system for the next 30 minutes before marking this incident as resolved.
Before: not present
After: We have received confirmation from our 3rd party billing provider that they are aware of their system outage and are working on a fix. We will resolve this incident when the 3rd party system is available.
Until this incident is marked as resolved, no billing functionality will be available, but all other functionality remains unaffected.
Before: not present
After: Our 3rd party billing system is now available and we are marking this incident as resolved. We will follow up with a post mortem update.
Before: not present
After: **Components impacted**
* Admin console
**Summary**
At 18:59 UTC we received an automated alert that requests to our 3rd party billing system \(Chargebee\) were failing. A few minutes later we confirmed that this was causing the Twingate admin console interface to fail to load with a 500 error, impacting all customers. At 19:15 UTC our engineering team submitted a hotfix to resolve the issue and update our production systems. The fix was deployed at 19:33 UTC and all customers were able to access the Twingate admin console with billing functionality disabled. We spoke to Chargebee at 20:00 UTC and they confirmed the issue on their side. At 20:47 UTC Chargebee confirmed that the outage was resolved in their system and the incident was closed.
**Root cause**
The Twingate admin console relies on Chargebee API access in order to load billing information specific to the particular customer account. This API call is made when the Twingate admin console loads in the browser. The underlying 3rd party API returned a 503 \(Service Temporarily Unavailable\) error, which was not captured as an exception in the error object returned by the 3rd party API library. This led to an uncaught exception, which caused the admin console to fail to load with a generic 500 \(Internal Server Error\) error.
**Corrective actions**
We have updated the admin console logic to capture this type of exception from Chargebee and ensure that the admin console will continue to load. We will be taking the following actions:
1. Auditing all 3rd party calls in the admin console to ensure that all exceptions are caught and do not result in the admin console being unavailable.
2. Update the billing behavior specifically to incorporate an unavailability message.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Wednesday, December 15, 2021
1 incident change
- MajorMajor incident
Admin console billing functionality is temporarily unavailablethis change