Companies / Lightdash / incidents

Lightdash: Degraded performance and Snowflake Query Failures

Impact: majorStatus: postmortemStarted Resolved (0 min)2 updatesSource: stspg.io (opens in new tab)
Summary

Lightdash reported a major incident affecting Lightdash Cloud (EU) and Lightdash Cloud (US), with Snowflake query failures, dashboards not loading charts, app crashes and high backend latency. It started and was marked resolved at 2023-10-03 15:39 UTC (recorded duration: 0 min); Lightdash said it identified the root cause and took corrective measures but provided no specifics.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Lightdash Cloud (EU)Lightdash Cloud - Lightdash Cloud (EU)Lightdash Cloud - Lightdash Cloud (US)Lightdash Cloud (US)

Timeline of updates

  1. resolved

    We've successfully identified the root cause and have taken measures to resolve the issue. We apologize for any inconvenience this may have caused and appreciate your understanding and patience during this time.

  2. postmortem

    #### 2:11 PM * **Issue Reported**: Users report that dashboards aren’t loading any charts. #### 3:41 PM * **Additional Reports**: Other users report that app is crashing #### 3:14 PM - 4:30 PM * **Observation**: High backend latency was detected. * **Collaboration**: Engaged in a chat with other engineers to identify potential causes. * **Root Cause Analysis**: * Reviewed changes released that morning. * Suspected the Node upgrade might be the cause. * Discovered a related GitHub issue with Snowflake and Node 20: [GitHub Issue](https://github.com/snowflakedb/snowflake-connector-nodejs/issues/588) * **Decision**: * Immediate rollback. * Plan to create a PR to update the `snowflake-sdk` package. #### 4:25 PM * **Rollback**: Rolled back from release 0.797.0 to 0.795.0. #### 4:47 PM * **Feedback**: Users report that app is functioning correctly #### 5:10 PM * **Follow-up Action**: Began working on the PR to upgrade Snowflake's SDK.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Lightdash profile · Recent Lightdash incidents · Markdown · JSON