Search

Companies / Metabase Cloud / incidents

Metabase Cloud: Issue with Metabase Cloud

Impact: criticalStatus: postmortemStarted Resolved (27 min)3 updatesSource: stspg.io (opens in new tab)
Summary

Metabase Cloud reported a critical outage blocking ingress network access to all instances, starting 2022-11-17 18:47 UTC and resolved at 2022-11-17 19:14 UTC (26 min). During new K8S cluster provisioning, DNS and load balancer configurations were accidentally overwritten; Metabase restored the correct configurations and added process controls and automated checks.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Metabase Cloud Platform

Timeline of updates

  1. investigating

    We identified an issue with our Cloud. We're currently investigating

  2. resolved

    We have restored the service. We will provide a description of the issue in the next few days.

  3. postmortem

    While provisioning new K8S clusters in our AWS environment, DNS and load balancer related configurations were accidentally overwritten with wrong parameters, preventing ingress network access to all Metabase Cloud instances. We have restored the correct configurations and added additional process controls and automated checks to prevent this from happening in the future.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Metabase Cloud profile · Recent Metabase Cloud incidents · Markdown · JSON