# Lightdash: Major incident: Intermittent API errors across US

**Summary:** Lightdash Cloud (US) experienced intermittent API request failures with major impact from 2026-03-24 10:35 UTC to 2026-03-24 12:19 UTC, lasting 1 h 43 min. Outbound connections exceeded cloud networking capacity, dropping requests; Lightdash restructured its internal network architecture to reduce outbound connection load and reported all services healthy after deploying the fix. _(machine-written, sol)_
- Kind: Major incident · Label: degradation (model (Jev))
- Changed: 2026-03-24 10:35 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/970vh708h991](https://stspg.io/970vh708h991)
- [Company: Lightdash](https://rewindex.com/company/lightdash.md)
- Versions: before (none: first archived) → after incident\_version:68/b0j060x9z0nf@0ba3465ff29d
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Intermittent API errors across US
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2026-03-24 10:35 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-03-24 12:19 UTC
```

**6. Added: Affected components**

```diff
+ Lightdash Cloud - Lightdash Cloud (US), Lightdash Cloud (US)
```

**7. Added: Update** (identified) · 2026-03-24 10:35 UTC

```diff
+ We have identified an issue in our network configuration that is leading to some requests failing. We've identified fix and are deploying to production.
```

**8. Added: Update** (identified) · 2026-03-24 10:35 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**9. Added: Update** (identified) · 2026-03-24 10:36 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**10. Added: Update** (monitoring) · 2026-03-24 11:34 UTC

```diff
+ We have deployed a fix to our network configuration and all services are healthy. We will continue monitoring.
```

**11. Added: Update** (resolved) · 2026-03-24 12:19 UTC

```diff
+ We've seen no errors related to network configuration.
```

**12. Added: Update** (postmortem) · 2026-03-24 12:21 UTC

```diff
+ **Root cause**   Our cloud networking layer had insufficient capacity to handle the volume of outbound connections from our services. As traffic grew, we periodically exceeded the available connection limits, causing some requests to be dropped.      **What we fixed**   We restructured our internal network architecture, significantly reducing the load on our outbound connection layer. This freed up capacity. ‌ **Ensuring it won’t happen again** We’ve permanently re-architected to prevent this specific issue. We’ve added monitoring \+ alerts to catch similar network errors.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-03-24 10:35 UTC | Major incident: Intermittent API errors across US (this change) | degradation |

All Lightdash changes: [timeline](https://rewindex.com/company/lightdash/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i846c3ee6e873c6 · Markdown: https://rewindex.com/change/i846c3ee6e873c6.md · JSON: https://rewindex.com/api/v1/changes/i846c3ee6e873c6 · Generated 2026-10-18T02:55:36.667Z