Dremio: Queries stuck in metadata retrieval in Dremio Cloud (EU)

Impact: majorStatus: resolvedStarted 2025-02-10 10:02 UTCResolved 2025-02-10 17:55 UTC (7.9 h)10 updatesSource: stspg.io

Summary

Dremio Cloud (EU) queries stalled at metadata retrieval during a major incident from 2025-02-10 10:02 UTC to 2025-02-10 17:55 UTC (7 h 53 min), now resolved. Dremio reported upstream timeouts and queuing in a core indexing service, mitigated by throttling; initial analysis suggested a recent underlying infrastructure change on Google may have triggered the failure.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Timeline of updates

investigating2025-02-10 10:02 UTC
We are currently investigating an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
investigating2025-02-10 10:55 UTC
We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
investigating2025-02-10 11:53 UTC
We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
investigating2025-02-10 12:52 UTC
We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
investigating2025-02-10 13:51 UTC
We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
investigating2025-02-10 14:57 UTC
We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
investigating2025-02-10 15:58 UTC
We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We are experiencing issues with our upstream provider causing timeouts and are working to mitigate this issue. We will provide another update within 60 minutes.
monitoring2025-02-10 17:15 UTC
We have identified the issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage and applied a mitigation. We are seeing recovery of customer environments and will continue to monitor. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.
monitoring2025-02-10 17:55 UTC
One of our core indexing services that powers both discovery of sources and metadata refreshes starting queueing up. We have mitigated the issue by throttling the service. Our initial analysis suggests that the failure may have been triggered by a recent change in our underlying infrastructure on Google. We are collaborating with our partners at Google to conduct a thorough Root Cause Analysis and identify the exact cause of the issue.
resolved2025-02-10 17:55 UTC
This incident has been resolved.

Rewindex changes for this incident

Major incident: Queries stuck in metadata retrieval in Dremio Cloud (EU) 2025-02-10 10:02 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Dremio profile · Recent Dremio incidents · Markdown · JSON