Search

Companies / Metabase / incidents

Metabase: Issue with Metabase Cloud

Impact: criticalStatus: postmortemStarted Resolved (27 min)3 updatesSource: stspg.io (opens in new tab)
Summary

Metabase Cloud suffered a critical outage blocking ingress network access to all instances, starting 2022-11-17 18:47 UTC and resolved at 2022-11-17 19:14 UTC (26 min). DNS and load balancer configurations were accidentally overwritten while provisioning new K8S clusters in AWS; Metabase restored the correct configurations and added process controls and automated checks.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Metabase Cloud Platform

Timeline of updates

  1. investigating

    We identified an issue with our Cloud. We're currently investigating

  2. resolved

    We have restored the service. We will provide a description of the issue in the next few days.

  3. postmortem

    While provisioning new K8S clusters in our AWS environment, DNS and load balancer related configurations were accidentally overwritten with wrong parameters, preventing ingress network access to all Metabase Cloud instances. We have restored the correct configurations and added additional process controls and automated checks to prevent this from happening in the future.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Metabase profile · Recent Metabase incidents · Markdown · JSON