Svix: Dashboard and application portal were temporarily inaccessible for some users
Timeline of updates
- resolved
The dashboard and application portal were temporarily inaccessible for ~15 minutes for some users. There was a configuration issue that was causing wrong TLS certificates to be used.
- postmortem
## Chain of events 1. At 11:51pm UTC a new deployment changed the certificate to include the wrong CNAMEs. 2. The team discovered it at 11:52pm UTC. 3. Found the cause and prepared a fix at 11:57pm UTC 4. Issue resolved \(TLS certificate propagated\) at 12:07am UTC. ## What went wrong 1. A change for an upcoming big feature inadvertently changed the certificates used by the servers serving our static content \(dashboard and app portal\). 2. Since AWS may cache old certificates for short period of time both our automatic and manual tests were served the good \(old\) certificates and passed. This happened both in staging and in production. 3. We have ongoing monitoring on the API endpoints in both staging and production - but for the dashboard and frontend only in production. ## Aftermath We have yet to conclude our full analysis though will draw our conclusions and put additional safety mechanisms in place to ensure this \(and similar incidents\) never happen again.
Changes Rewindex caught on this incident
- Aug 5, 2021Major incident: Dashboard and application portal were temporarily inaccessible for some users
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).