Changes / Sigma Computing / Incorrect timestamps returned from warehouse for some customers
Sigma Computing: Major incident: Incorrect timestamps returned from warehouse for some customers
Summary
Sigma Computing reported a major incident affecting some customers: warehouse timestamps were off by their UTC offset, and date_trunc() could return incorrect values due to code refactoring. The recorded start is 2023-08-14 17:34 UTC, but resolution is dated earlier, 2023-08-07 17:32 UTC; duration is unknown.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 75%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Incorrect timestamps returned from warehouse for some customers
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-08-14 17:34 UTC
Before: not present
After: 2023-08-07 17:32 UTC
Before: not present
After: Backdated Incident: Some customers observed that timestamps in data returned from the warehouse were incorrect. Specifically the timestamp was off by the UTC offset (hours the customer timezone is ahead or behind UTC). Customers may have also observed that date_trunc() would return incorrect values. This was due to a change made during a code refactoring; no behavioral difference was anticipated, but certain customer configurations experienced the described discrepancy.
Impacted Cloud Regions & Services: Snowflake and DataBricks customers with a particular combination of timezone configuration in Sigma and their warehouse were impacted by this issue.
Note the refactored code path was enabled for some customers on August 7, and then progressively enabled for more customers and warehouses over several days; notably:
- August 8, 17:30 UTC, code path enabled for all Databricks customers
- August 10, 15:58 UTC, code path enabled for all Snowflake customers
- August 10, 18:49 UTC, code path disabled for all Snowflake customers
- August 14, 15:40 UTC, code path disabled for all customers
Incident Start Time: 2023-08-07 18:10 UTC
Incident End Time: 2023-08-14 15:40 UTC
We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days.
Before: not present
After: Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis \(RCA\) is available on the Sigma Community site: [Postmortem for Aug 10th, 2023: Incorrect results for timezone-based calculations](https://community.sigmacomputing.com/t/postmortem-for-aug-10th-2023-incorrect-results-for-timezone-based-calculations/2618)
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Incorrect timestamps returned from warehouse for some customers (this change) Sigma Computing reported a major incident affecting some customers: warehouse timestamps were off by their UTC offset, and date_trunc() could return incorrect values due to code refactoring. The recorded start is 2023-08-14 17:34 UTC, but resolution is dated earlier, 2023-08-07 17:32 UTC; duration is unknown. | Major incident |
All Sigma Computing changes → · Sigma Computing profile · Markdown · JSON