Changes / Metabase Cloud / Issue with SSL certificates
Metabase Cloud: Critical incident: Issue with SSL certificates
Summary
Metabase Cloud reported a critical certificate outage starting 2023-11-09 19:37 UTC, lasting 24 min. Hosted-instance users received invalid certificate errors after a Helm chart release CI misconfiguration replaced the external Nginx ingress certificate with a dummy certificate. Certificates were restored by 2023-11-09 20:02 UTC.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Issue with SSL certificates
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2023-11-09 19:37 UTC
Before: not present
After: 2023-11-09 20:02 UTC
Before: not present
After: Metabase Cloud Platform
Before: not present
After: We're investigating an issue in our cloud with certificate management that's causing browsers to receive incorrect certificates
Before: not present
After: Ingress controller certificates are now back to normal. You should be able to get the correct certificate in the browser now. We're very sorry for the inconvenience, we'll publish a retrospective about the issue in the next few days.
Before: not present
After: **Summary**
On Thursday Nov 9th the default certificate for the external Nginx ingress was replaced with a dummy certificate. Users attempting to reach their hosted instances would get an invalid certificate error. This was due to a misconfiguration in the new CI for releasing our helm charts that used the internal Nginx ingress values files for the external Nginx ingress.
**Impact**
All hosted customers not using custom domains were affected.
**How was the root cause diagnosed?**
We identified that the values were properly set in the values files. We Investigated the CI steps to determine why the correct values were not set. It was determined that the wrong values files were being used for external Nginx ingress.
**How we’ll make this not happen again?**
* Update staging and production value files to match.
* Add alert to pagerduty for non-custom domains \(catch alerts faster\)
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Issue with SSL certificates (this change) Metabase Cloud reported a critical certificate outage starting 2023-11-09 19:37 UTC, lasting 24 min. Hosted-instance users received invalid certificate errors after a Helm chart release CI misconfiguration replaced the external Nginx ingress certificate with a dummy certificate. Certificates were restored by 2023-11-09 20:02 UTC. | Major incident |
All Metabase Cloud changes → · Metabase Cloud profile · Markdown · JSON