Changes / Metabase Cloud / Degraded service in our Business tier
Metabase Cloud: Major incident: Degraded service in our Business tier
Summary
Metabase Cloud reported major degradation affecting Business tier customers in isolated environments from 2021-08-10 16:37 UTC to 2021-08-10 20:38 UTC, lasting 4 h 1 min. A failed Elastic Beanstalk upgrade left instances inconsistent, with load balancers unable to mark them healthy and application servers continuously recycled; a fix was deployed before resolution.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 65%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Degraded service in our Business tier
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2021-08-10 16:37 UTC
Before: not present
After: 2021-08-10 20:38 UTC
Before: not present
After: Metabase Cloud Platform
Before: not present
After: We identified an issue in our Business tier customers that run on isolated environments. The issue appears to be an issue with our load balancers not being able to flag instances as healthy which results in application servers being recycled continously.
Before: not present
After: We've identified a fix for the outage and we're in the process of issuing an update to resolve the situation
Before: not present
After: We've deployed a fix and we're monitoring the health of the instances.
Before: not present
After: This incident has been resolved.
Before: not present
After: A failure in the upgrade in our Elastic Beanstalk deployment for improving the timeout on reverse proxies led to instances not being able to revert back to previous version leaving them in an inconsistent state.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Degraded service in our Business tier (this change) Metabase Cloud reported major degradation affecting Business tier customers in isolated environments from 2021-08-10 16:37 UTC to 2021-08-10 20:38 UTC, lasting 4 h 1 min. A failed Elastic Beanstalk upgrade left instances inconsistent, with load balancers unable to mark them healthy and application servers continuously recycled; a fix was deployed before resolution. | Major incident |
All Metabase Cloud changes → · Metabase Cloud profile · Markdown · JSON