Companies / Sigma Computing / incidents
Sigma Computing: Materialization failures impacting GCP cloud-hosted orgs
Sigma reported a major incident causing materialization and CSV upload failures for GCP-hosted customers, including GCP (US). Recorded from 2023-09-06 20:24 UTC to 2023-09-06 21:16 UTC, it lasted 52 min. Mitigation resolved the issue; affected materializations were expected to recover on their next scheduled run or when manually retriggered.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: GCP - Sigma - GCP (US)Sigma - GCP (US)
Timeline of updates
- investigating
Some customers hosted on Google Cloud Platform (GCP) have observed Materialization and CSV upload failures beginning at 1:00pm EST / 10:00am PST. Our Engineering team is currently conducting an investigation to identify and mitigate the root cause, and Technical Support is actively monitoring the situation. We are currently investigating an issue with materialization and CSV upload failure for customers on Google Cloud Platform (GCP). This issue is being treated with the highest priority and we will update the status accordingly. Impacted Cloud Regions & Services: Sigma organizations hosted on GCP utilizing materialization User Experience: Errors related to materialization and CSV upload failure Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
- monitoring
Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are monitoring their effect closely. Impacted Cloud Regions & Services: Sigma users hosted on GCP User Experience: We expect the rate of errors for affected materializations to decrease on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP. Incident Start Time: Approximately 17:00 UTC Sep 6, 2023 Incident End Time: Approximately 19:40 UTC Sep 6, 2023
- resolved
Current Status: The mitigation measures have proven effective, and the issue is resolved. Impacted Cloud Regions & Services: Sigma users hosted on GCP User Experience: We expect affected materializations to be resolved on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP. Incident Start Time: Approximately 16:30 UTC Sep 6, 2023 Incident End Time: Approximately 19:40 UTC Sep 6, 2023 We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
- postmortem
Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis \(RCA\) is available on the Sigma Community site: [Post Mortem for Sep 6, 2023: Materialization failures impacting GCP cloud-hosted orgs](https://community.sigmacomputing.com/t/post-mortem-for-sep-6-2023-materialization-failures-impacting-gcp-cloud-hosted-orgs/2600).
Rewindex changes for this incident
- Major incident: Materialization failures impacting GCP cloud-hosted orgs 2023-09-06 20:24 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Sigma Computing profile · Recent Sigma Computing incidents · Markdown · JSON