Search

Companies / Lightdash / incidents

Lightdash: app.lightdash.cloud showing degraded performance

Impact: minorStatus: postmortemStarted Resolved (3.6 h)6 updatesSource: stspg.io (opens in new tab)

Affected components: Lightdash Cloud - Lightdash Cloud (US)Lightdash Cloud (US)

Timeline of updates

  1. investigating

    We are currently investigating the root cause of the issue

  2. investigating

    We have identified the failing component but are still finding the root cause. Services appear operational again.

  3. identified

    We've identified the root cause and it's being fixed.

  4. monitoring

    A fix has been implemented and we are monitoring the deployment.

  5. resolved

    This incident is resolved.

  6. postmortem

    Earlier today we received an automated alert that app.lightdash.cloud was unavailable and returning 502 errors. The reason for this error was that Lightdash was slowing down due to the amount of usage in Lightdash Cloud at the time. The slower response times in Lightdash triggered an automated process to restart the Lightdash servers, usually this should only trigger in the case that the server has already crashed. In this incident, this was a mistake and the server was simply running more slowly than expected. To resolve the issue, we've added much more resource to our Lightdash Cloud servers to prevent slow response times. We've also increased the threshold to automatically restarting the servers in the case of very slow response times.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Lightdash profile · Recent Lightdash incidents · Markdown · JSON