Changes / Sigma Computing / Materialization failures impacting GCP cloud-hosted orgs
Sigma Computing: Major incident: Materialization failures impacting GCP cloud-hosted orgs
Summary
Sigma reported a major incident causing materialization and CSV upload failures for GCP-hosted customers, including GCP (US). Recorded from 2023-09-06 20:24 UTC to 2023-09-06 21:16 UTC, it lasted 52 min. Mitigation resolved the issue; affected materializations were expected to recover on their next scheduled run or when manually retriggered.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Materialization failures impacting GCP cloud-hosted orgs
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-09-06 20:24 UTC
Before: not present
After: 2023-09-06 21:16 UTC
Before: not present
After: GCP - Sigma - GCP (US), Sigma - GCP (US)
Before: not present
After: Some customers hosted on Google Cloud Platform (GCP) have observed Materialization and CSV upload failures beginning at 1:00pm EST / 10:00am PST. Our Engineering team is currently conducting an investigation to identify and mitigate the root cause, and Technical Support is actively monitoring the situation.
We are currently investigating an issue with materialization and CSV upload failure for customers on Google Cloud Platform (GCP). This issue is being treated with the highest priority and we will update the status accordingly.
Impacted Cloud Regions & Services: Sigma organizations hosted on GCP utilizing materialization
User Experience: Errors related to materialization and CSV upload failure
Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
Before: not present
After: Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are monitoring their effect closely.
Impacted Cloud Regions & Services: Sigma users hosted on GCP
User Experience: We expect the rate of errors for affected materializations to decrease on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP.
Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
Incident End Time: Approximately 19:40 UTC Sep 6, 2023
Before: not present
After: Current Status: The mitigation measures have proven effective, and the issue is resolved.
Impacted Cloud Regions & Services: Sigma users hosted on GCP
User Experience: We expect affected materializations to be resolved on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP.
Incident Start Time: Approximately 16:30 UTC Sep 6, 2023
Incident End Time: Approximately 19:40 UTC Sep 6, 2023
We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
Before: not present
After: Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis \(RCA\) is available on the Sigma Community site: [Post Mortem for Sep 6, 2023: Materialization failures impacting GCP cloud-hosted orgs](https://community.sigmacomputing.com/t/post-mortem-for-sep-6-2023-materialization-failures-impacting-gcp-cloud-hosted-orgs/2600).
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Materialization failures impacting GCP cloud-hosted orgs (this change) Sigma reported a major incident causing materialization and CSV upload failures for GCP-hosted customers, including GCP (US). Recorded from 2023-09-06 20:24 UTC to 2023-09-06 21:16 UTC, it lasted 52 min. Mitigation resolved the issue; affected materializations were expected to recover on their next scheduled run or when manually retriggered. | Major incident |
All Sigma Computing changes → · Sigma Computing profile · Markdown · JSON