# Convex: Major incident: Convex traffic having downtime

**Summary:** Convex reported a major incident causing intermittent unavailability across all Convex services, starting 2025-09-20 12:18 UTC and resolved at 2025-09-20 13:15 UTC (57 min). The postmortem cited a cascading traffic-layer failure after a Caddy node ran out of memory during an unforeseen load spike; Convex added capacity. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2025-09-20 12:18 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/ypq0hqy4gmc2](https://stspg.io/ypq0hqy4gmc2)
- [Company: Convex](https://rewindex.com/company/convex.md)
- Versions: before (none: first archived) → after incident\_version:140/qnc3pbv0wnjs@fe638e21326d
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Convex traffic having downtime
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2025-09-20 12:18 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-09-20 13:15 UTC
```

**6. Added: Update** (investigating) · 2025-09-20 12:18 UTC

```diff
+ We are currently investigating this issue.
```

**7. Added: Update** (monitoring) · 2025-09-20 12:57 UTC

```diff
+ An unexpected traffic pattern overloaded some of our services, causing intermittent unavailability across Convex instances. We've added extra capacity and are monitoring to ensure that the system is stable.
```

**8. Added: Update** (resolved) · 2025-09-20 13:15 UTC

```diff
+ This incident has been resolved.
```

**9. Added: Update** (postmortem) · 2025-09-20 20:42 UTC

```diff
+ From around 5:18am to 5:54am Pacific \(12:18pm to 12:54pm UTC\), Convex had a 36 min period of intermittent downtime that affected all Convex services.      The specific issue was a cascading failure in our traffic layer. We had a traffic node \(Caddy\) run out of memory due to an unforeseen load spike and instead of just being restarted/replaced this node was marked as permanently down by our container management layer \(Nomad\) which led to the issue propagating to all traffic servers.      Since the incident we've more than doubled the size of our traffic layer, fixed the failover behavior which led to nodes staying failed after OOMing, and will be investigating alternative traffic services.      As always data was safe during this incident but we really apologize for the availability impact during that time period.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-09-20 12:18 UTC | Major incident: Convex traffic having downtime (this change) | outage |

All Convex changes: [timeline](https://rewindex.com/company/convex/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i388b7a9960a312 · Markdown: https://rewindex.com/change/i388b7a9960a312.md · JSON: https://rewindex.com/api/v1/changes/i388b7a9960a312 · Generated 2026-10-18T04:44:36.023Z