Changes / Dremio / Query engines not scaling up
Dremio: Major incident: Query engines not scaling up
Summary
Dremio Cloud - US had a major incident: query engines failed to scale up on demand, causing query failures; already-running engines remained unaffected. It started at 2022-05-10 05:32 UTC and was resolved at 2022-05-10 06:49 UTC, lasting 1 h 16 min. Dremio reported a successful mitigation and confirmed engine scaling was working normally.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Query engines not scaling up
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2022-05-10 05:32 UTC
Before: not present
After: 2022-05-10 06:49 UTC
Before: not present
After: Dremio Cloud - US - Site Availability, Site Availability
Before: not present
After: Query engines are not scaling up on demand leading to query failures.
Engines that are already running will continue to server queries without issue.
We are currently investigating the engine scaling issue.
Before: not present
After: We are continuing to investigate the issue. One potential mitigation has been applied.
We are evaluating the effectiveness of the change.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: A mitigation has been applied and appears successful.
Monitoring for rebound.
Before: not present
After: This incident has been resolved.
Engine scaling working normally.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Query engines not scaling up (this change) Dremio Cloud - US had a major incident: query engines failed to scale up on demand, causing query failures; already-running engines remained unaffected. It started at 2022-05-10 05:32 UTC and was resolved at 2022-05-10 06:49 UTC, lasting 1 h 16 min. Dremio reported a successful mitigation and confirmed engine scaling was working normally. | Major incident |