Changes / Dremio / Query failures in Dremio Cloud (US & EU)
Dremio: Major incident: Query failures in Dremio Cloud (US & EU)
Summary
Dremio Cloud (US & EU) had a major incident causing query failures when using reflections, starting 2025-11-11 21:51 UTC and resolved 2025-11-12 07:13 UTC (9 h 21 min). Dremio attributed the disruption to an internal-source permissions update restricting access to some project-store folders and reported deploying a fix.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Query failures in Dremio Cloud (US & EU)
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-11-11 21:51 UTC
Before: not present
After: 2025-11-12 07:13 UTC
Before: not present
After: Dremio Cloud - Europe - Site Availability, Dremio Cloud - US - Site Availability, Site Availability
Before: not present
After: We are investigating issues with queries in Dremio Cloud (US & EU). We will provide an update within the next 30 minutes.
Before: not present
After: We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate the issue and will provide an update within the next 30 minutes.
Before: not present
After: We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate and narrow down the issue. We will provide an update within the next 30 minutes.
Before: not present
After: We are continuing to investigate the issue with queries in Dremio Cloud (US & EU). Engineering has narrowed down the issue and are working on multiple possible mitigations. We will provide an update within the next 45 minutes.
Before: not present
After: We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Before: not present
After: We have narrowed down the issue impacting queries in Dremio Cloud (US & EU). Engineering is in the process of building and testing a fix to address the issue. We will provide an update within the next 45 minutes.
Before: not present
After: We have identified the issue impacting queries in Dremio Cloud (US & EU) and are in the process of deploying a fix to address the issue. We will provide an update within the next 45 minutes.
Before: not present
After: Incident Summary
We experienced a service disruption starting at 1:15 PM PST that resulted in query failures when using reflections in multiple regions. We sincerely apologize for the impact this had on your workflows and the inconvenience it may have caused.
Incident Details
An update to the permissions used by Dremio internal sources inadvertently restricted access to some of the folders in the project store, resulting in queries using reflections or accessing those folders to fail in some AWS projects under certain conditions. Our team began investigating the issue immediately and was able to identify the issue and implement a fix at 5 PM PST. Concurrently, we encountered cloud provider issues in our Azure deployment infrastructure which impacted our ability to roll out the fix to our production environments. Once we were able to resolve issue, we were able to validate and deploy a fix to fully restore the service at 10:35 PM PST.
Our engineers are monitoring the system to ensure full stability. In addition, we are also implementing new alerts to detect such situation quickly in the future.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Query failures in Dremio Cloud (US & EU) (this change) Dremio Cloud (US & EU) had a major incident causing query failures when using reflections, starting 2025-11-11 21:51 UTC and resolved 2025-11-12 07:13 UTC (9 h 21 min). Dremio attributed the disruption to an internal-source permissions update restricting access to some project-store folders and reported deploying a fix. | Major incident |