Changes / Dremio / Intermittent Query Failures
Dremio: Major incident: Intermittent Query Failures
Summary
Dremio reported major intermittent query failures starting 2023-09-07 15:50 UTC, believed limited to AWS projects created since Aug 17th, 2023, following a deployment. Fixes restored normal operation in US and European production environments; the incident was resolved at 2023-09-08 17:40 UTC after 25 h 50 min.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 55%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Intermittent Query Failures
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2023-09-07 15:50 UTC
Before: not present
After: 2023-09-08 17:40 UTC
Before: not present
After: Dremio Cloud - Europe - Site Availability, Dremio Cloud - US - Site Availability, Site Availability
Before: not present
After: Following the deployment on Wednesday, Sept 6th, 2023 our monitoring has detected an increase in query failure rates for some projects. We believe this is only impacting AWS projects created since Aug 17th, 2023.
If you are experiencing query failures, please contact Dremio Support and they can assist in mitigation.
We believe the issue has been identified and a fix is being evaluated. We will update further when the fix is deployed.
Before: not present
After: The issue has been identified and a fix is being implemented.
Before: not present
After: We are moving the mitigation through our testing environments in preparation of deploying to our production systems.
Before: not present
After: A fix has been deployed to the US site and is being evaluated. Initial reports are showing increased latency in the UI. We will update as we consider next steps.
Before: not present
After: The fix deployed to the US site is performing well now. We will continue to monitor and then roll out the same fix to the European site.
Before: not present
After: The fix has been deployed to both production environments and normal operation has resumed. We will continue to monitor before resolving the incident.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Intermittent Query Failures (this change) Dremio reported major intermittent query failures starting 2023-09-07 15:50 UTC, believed limited to AWS projects created since Aug 17th, 2023, following a deployment. Fixes restored normal operation in US and European production environments; the incident was resolved at 2023-09-08 17:40 UTC after 25 h 50 min. | Major incident |