Changes / Dremio / Queries stuck in metadata retrieval in Dremio Cloud (EU)
Dremio: Major incident: Queries stuck in metadata retrieval in Dremio Cloud (EU)
Summary
Dremio Cloud (EU) queries stalled at metadata retrieval during a major incident from 2025-02-10 10:02 UTC to 2025-02-10 17:55 UTC (7 h 53 min), now resolved. Dremio reported upstream timeouts and queuing in a core indexing service, mitigated by throttling; initial analysis suggested a recent underlying infrastructure change on Google may have triggered the failure.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Queries stuck in metadata retrieval in Dremio Cloud (EU)
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-02-10 10:02 UTC
Before: not present
After: 2025-02-10 17:55 UTC
Before: not present
After: We are currently investigating an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
Before: not present
After: We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
Before: not present
After: We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
Before: not present
After: We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
Before: not present
After: We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
Before: not present
After: We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
Before: not present
After: We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We are experiencing issues with our upstream provider causing timeouts and are working to mitigate this issue. We will provide another update within 60 minutes.
Before: not present
After: We have identified the issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage and applied a mitigation. We are seeing recovery of customer environments and will continue to monitor. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.
Before: not present
After: One of our core indexing services that powers both discovery of sources and metadata refreshes starting queueing up. We have mitigated the issue by throttling the service.
Our initial analysis suggests that the failure may have been triggered by a recent change in our underlying infrastructure on Google. We are collaborating with our partners at Google to conduct a thorough Root Cause Analysis and identify the exact cause of the issue.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Queries stuck in metadata retrieval in Dremio Cloud (EU) (this change) Dremio Cloud (EU) queries stalled at metadata retrieval during a major incident from 2025-02-10 10:02 UTC to 2025-02-10 17:55 UTC (7 h 53 min), now resolved. Dremio reported upstream timeouts and queuing in a core indexing service, mitigated by throttling; initial analysis suggested a recent underlying infrastructure change on Google may have triggered the failure. | Major incident |