Changes / Lightdash / Lightdash Cloud instability (slow response times)
Lightdash: Major incident: Lightdash Cloud instability (slow response times)
Summary
Lightdash Cloud (US) experienced major API degradation affecting all users, with actions taking minutes or not executing. It started at 2023-05-09 13:48 UTC and was resolved at 2023-05-09 15:33 UTC; the computed duration was 1 h 44 min. Lightdash reported increasing server count and size as its initial response.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Lightdash Cloud instability (slow response times)
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-05-09 13:48 UTC
Before: not present
After: 2023-05-09 15:33 UTC
Before: not present
After: Lightdash Cloud - Lightdash Cloud (US), Lightdash Cloud (US)
Before: not present
After: The Lightdash API a https://app.lightdash.cloud is experiencing very slow response times for all users, leading to some actions taking minutes or not executing.
We're currently investigating.
Before: not present
After: We are monitoring performance
Before: not present
After: This incident has been resolved.
Before: not present
After: Today at 13:37 \(UTC\) we noticed that all API endpoints for [https://app.lightdash.cloud](https://app.lightdash.cloud) were starting to run very slow. This was affecting all users and led to response times of over a minute for the API.
Our first response was to greatly increase the amount of resources available for the Lightdash server \(both the number and size of servers\). After this change all services remained stable.
We have investigated our logs to understand exactly what actions user’s were taking to make the server so busy. In addition to the volume of users, we noticed a higher number than usual of Databricks users. We already have a fix in testing for improved Databricks performance, which can be expected in the coming hours. For further performance improvements you can follow this milestone in GitHub: [https://github.com/lightdash/lightdash/milestone/91](https://github.com/lightdash/lightdash/milestone/91)
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Lightdash Cloud instability (slow response times) (this change) Lightdash Cloud (US) experienced major API degradation affecting all users, with actions taking minutes or not executing. It started at 2023-05-09 13:48 UTC and was resolved at 2023-05-09 15:33 UTC; the computed duration was 1 h 44 min. Lightdash reported increasing server count and size as its initial response. | Major incident |
All Lightdash changes → · Lightdash profile · Markdown · JSON