Changes / Lightdash / Intermittent API errors across US
Lightdash: Major incident: Intermittent API errors across US
Summary
Lightdash Cloud (US) experienced intermittent API request failures with major impact from 2026-03-24 10:35 UTC to 2026-03-24 12:19 UTC, lasting 1 h 43 min. Outbound connections exceeded cloud networking capacity, dropping requests; Lightdash restructured its internal network architecture to reduce outbound connection load and reported all services healthy after deploying the fix.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 40%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Intermittent API errors across US
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-03-24 10:35 UTC
Before: not present
After: 2026-03-24 12:19 UTC
Before: not present
After: Lightdash Cloud - Lightdash Cloud (US), Lightdash Cloud (US)
Before: not present
After: We have identified an issue in our network configuration that is leading to some requests failing. We've identified fix and are deploying to production.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: We have deployed a fix to our network configuration and all services are healthy. We will continue monitoring.
Before: not present
After: We've seen no errors related to network configuration.
Before: not present
After: **Root cause**
Our cloud networking layer had insufficient capacity to handle the volume of outbound connections from our services. As traffic grew, we periodically exceeded the available connection limits, causing some requests to be dropped.
**What we fixed**
We restructured our internal network architecture, significantly reducing the load on our outbound connection layer. This freed up capacity.
**Ensuring it won’t happen again**
We’ve permanently re-architected to prevent this specific issue. We’ve added monitoring \+ alerts to catch similar network errors.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Intermittent API errors across US (this change) Lightdash Cloud (US) experienced intermittent API request failures with major impact from 2026-03-24 10:35 UTC to 2026-03-24 12:19 UTC, lasting 1 h 43 min. Outbound connections exceeded cloud networking capacity, dropping requests; Lightdash restructured its internal network architecture to reduce outbound connection load and reported all services healthy after deploying the fix. | Major incident |
All Lightdash changes → · Lightdash profile · Markdown · JSON