# Convex: Critical incident: high error rates on customer backends

**Summary:** Convex reported a critical incident causing high backend error rates and downtime for some customers, affecting Free & Starter and Live Traffic database, function and Node.js runtimes. It started 2026-03-04 17:28 UTC and lasted 1 h 0 min. Convex remediated the issue and attributed it to a load spike triggering load-shedding and a caching-library panic. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-03-04 17:28 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/wqspk7fqxyw5](https://stspg.io/wqspk7fqxyw5)
- [Company: Convex](https://rewindex.com/company/convex.md)
- Versions: before (none: first archived) → after incident\_version:140/0wmzc599wplz@5d232da4356f
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ high error rates on customer backends
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2026-03-04 17:28 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-03-04 18:28 UTC
```

**6. Added: Affected components**

```diff
+ Free & Starter, Live Traffic - Database Services, Live Traffic - Function Runtime, Live Traffic - Node.js Runtime
```

**7. Added: Update** (investigating) · 2026-03-04 17:28 UTC

```diff
+ We are currently investigating an issue leading to elevated error rates on customer backends
```

**8. Added: Update** (monitoring) · 2026-03-04 18:14 UTC

```diff
+ We've identified the issue and remediated the problem. We're monitoring before we declare the all clear.
```

**9. Added: Update** (resolved) · 2026-03-04 18:28 UTC

```diff
+ Incident resolved. We're very sorry for the impact on your projects. Our team will be publishing a detailed postmortem soon.
```

**10. Added: Update** (postmortem) · 2026-03-05 18:14 UTC

```diff
+ Convex had downtime for some customers caused by an anomalous load spike that pushed an internal service into a load-shedding mode, which then triggered an unexpected panic in a caching library we were using. Specifically the load triggered two queue management algorithms: CoDel which proactively drops requests to keep queues small, and adaptive-LIFO which dequeues in reverse order to avoid wasting time on old requests. These are both rather subtle algorithms that large services use to avoid congestion collapse under high load or attack. The panic in the caching library was just a bug that depended on both these algorithms simultaneously. We've made some changes as a result of this incident but the key lesson is that services should try to avoid switching logical behavior during high load. When systems are stressed switching to infrequently-used codepaths can often make matters worse. We're now going to be proactively triggering CoDel and adaptive-LIFO at steady state to ensure that we're exercising this worst-case flow at all times. We apologize to our customers for impacting your products and services. We’re focusing intensely over the next few weeks on hardening our systems to prevent issues like this from happening again.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-03-04 17:28 UTC | Critical incident: high error rates on customer backends (this change) | outage |

All Convex changes: [timeline](https://rewindex.com/company/convex/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i67a18d82c57f8a · Markdown: https://rewindex.com/change/i67a18d82c57f8a.md · JSON: https://rewindex.com/api/v1/changes/i67a18d82c57f8a · Generated 2026-10-18T03:21:52.979Z