ChangesTwingateadmin console and authentication incident
Twingate: Critical incident: admin console and authentication incident
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Twingate removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: admin console and authentication incident
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2023-06-07 17:42 UTC
Before: not present
After: 2023-06-07 17:51 UTC
Before: not present
After: Admin Console, Admin Console - Admin Web Interface
Before: not present
After: We are seeing issues with our admin consoles not loading properly. It's been investigated.
Before: not present
After: This incident has been resolved. It's due to an issue with a software update. We are working on a plan to avoid this in the future. This impacted authentication flow too. Already authenticated flows continued to function.
Before: not present
After: **Summary**
On June 7th 17:08 UTC, a new version of our Controller software was rolled out. Shortly after the rollout completed, our on-call team received automated exception alerts and began investigating. The new version had inadvertently included changes to clean up our database that were out of sync with the deployed code, and the team decided to rollback the new software deployment. The rollback proceeded smoothly, however the previous code version was missing the database fields that had been cleaned up, and the incident started at 17:24 UTC.
A fix was prepared and rolled out starting at 17:29 UTC. Deployment completed on the first cluster at 17:38 UTC and proceeded to the remaining clusters once we verified that the error state had been resolved. Deployment was completed on all clusters at 17:48 UTC.
**Post-incident Analysis**
We initially thought the incident had only impacted Admin Console users, however the following systems were impacted:
* Admin Console sign in.
* Client initial authentication and re-authentication requests.
* Linux and container-based Connectors. This incident exposed an issue that resulted in Connectors incorrectly shutting down on transient Controller unavailability.
**Root Cause**
An error in our deployment process logic led to a mismatch in deployed code and database schema.
**Corrective Actions**
1. Improve our processes for merging software changes that are linked to database schema changes.
2. We have fixed and will be testing the bug in our Connector uptime / retry logic.
3. Improve overall build and rollout performance to be able to push fixes more promptly.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Wednesday, June 7, 2023
1 incident change
- CriticalCritical incident
admin console and authentication incidentthis change