Changes / Lightdash / Degraded performance and Snowflake Query Failures
Lightdash: Major incident: Degraded performance and Snowflake Query Failures
Summary
Lightdash reported a major incident affecting Lightdash Cloud (EU) and Lightdash Cloud (US), with Snowflake query failures, dashboards not loading charts, app crashes and high backend latency. It started and was marked resolved at 2023-10-03 15:39 UTC (recorded duration: 0 min); Lightdash said it identified the root cause and took corrective measures but provided no specifics.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Degraded performance and Snowflake Query Failures
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-10-03 15:39 UTC
Before: not present
After: 2023-10-03 15:39 UTC
Before: not present
After: Lightdash Cloud (EU), Lightdash Cloud - Lightdash Cloud (EU), Lightdash Cloud - Lightdash Cloud (US), Lightdash Cloud (US)
Before: not present
After: We've successfully identified the root cause and have taken measures to resolve the issue. We apologize for any inconvenience this may have caused and appreciate your understanding and patience during this time.
Before: not present
After: #### 2:11 PM
* **Issue Reported**: Users report that dashboards aren’t loading any charts.
#### 3:41 PM
* **Additional Reports**: Other users report that app is crashing
#### 3:14 PM - 4:30 PM
* **Observation**: High backend latency was detected.
* **Collaboration**: Engaged in a chat with other engineers to identify potential causes.
* **Root Cause Analysis**:
* Reviewed changes released that morning.
* Suspected the Node upgrade might be the cause.
* Discovered a related GitHub issue with Snowflake and Node 20: [GitHub Issue](https://github.com/snowflakedb/snowflake-connector-nodejs/issues/588)
* **Decision**:
* Immediate rollback.
* Plan to create a PR to update the `snowflake-sdk` package.
#### 4:25 PM
* **Rollback**: Rolled back from release 0.797.0 to 0.795.0.
#### 4:47 PM
* **Feedback**: Users report that app is functioning correctly
#### 5:10 PM
* **Follow-up Action**: Began working on the PR to upgrade Snowflake's SDK.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Degraded performance and Snowflake Query Failures (this change) Lightdash reported a major incident affecting Lightdash Cloud (EU) and Lightdash Cloud (US), with Snowflake query failures, dashboards not loading charts, app crashes and high backend latency. It started and was marked resolved at 2023-10-03 15:39 UTC (recorded duration: 0 min); Lightdash said it identified the root cause and took corrective measures but provided no specifics. | Major incident |
All Lightdash changes → · Lightdash profile · Markdown · JSON