ChangesSvixDashboard and application portal were temporarily inaccessible for some users
Svix: Major incident: Dashboard and application portal were temporarily inaccessible for some users
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Svix removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Dashboard and application portal were temporarily inaccessible for some users
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2021-08-05 00:07 UTC
Before: not present
After: 2021-08-05 00:07 UTC
Before: not present
After: The dashboard and application portal were temporarily inaccessible for ~15 minutes for some users.
There was a configuration issue that was causing wrong TLS certificates to be used.
Before: not present
After: ## Chain of events
1. At 11:51pm UTC a new deployment changed the certificate to include the wrong CNAMEs.
2. The team discovered it at 11:52pm UTC.
3. Found the cause and prepared a fix at 11:57pm UTC
4. Issue resolved \(TLS certificate propagated\) at 12:07am UTC.
## What went wrong
1. A change for an upcoming big feature inadvertently changed the certificates used by the servers serving our static content \(dashboard and app portal\).
2. Since AWS may cache old certificates for short period of time both our automatic and manual tests were served the good \(old\) certificates and passed. This happened both in staging and in production.
3. We have ongoing monitoring on the API endpoints in both staging and production - but for the dashboard and frontend only in production.
## Aftermath
We have yet to conclude our full analysis though will draw our conclusions and put additional safety mechanisms in place to ensure this \(and similar incidents\) never happen again.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Thursday, August 5, 2021
1 incident change
- MajorMajor incident
Dashboard and application portal were temporarily inaccessible for some usersthis change