# Convex: Major incident: Small percentage of deployments experiencing outage

**Summary:** A small percentage of Convex deployments experienced sporadic availability in a major-impact outage from 2026-04-09 05:42 UTC to 2026-04-09 06:24 UTC (42 min). During database primary failover, orchestration failed to provision a replica ready for new writes; Convex implemented a fix and reported resolution. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-04-09 05:42 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/byp02y8zrsky](https://stspg.io/byp02y8zrsky)
- [Company: Convex](https://rewindex.com/company/convex.md)
- Versions: before (none: first archived) → after incident\_version:140/sckl653cwdx1@c3994072d7f4
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Small percentage of deployments experiencing outage
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-04-09 05:42 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-04-09 06:24 UTC
```

**6. Added: Affected components**

```diff
+ Business, Development Services - Business, Development Services - Free & Starter, Development Services - Professional, Free & Starter, Live Traffic - Business, Live Traffic - Free & Starter, Live Traffic - Professional, Professional
```

**7. Added: Update** (investigating) · 2026-04-09 05:42 UTC

```diff
+ Seems related to a problematic database failover. Updates soon.
```

**8. Added: Update** (monitoring) · 2026-04-09 05:59 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**9. Added: Update** (resolved) · 2026-04-09 06:24 UTC

```diff
+ This issue is considered resolved. Impact: A small percentage of Convex deployments experienced sporadic availability for approximately 17 minutes, from 22:42 to 22:59 PM PT. Root cause: one of our database clusters experienced a routine primary failover event. During the primary promotion process, the orchestration system failed to provision a replica that was ready to accept new database writes. Out of an abundance of caution, Convex will only use database clusters that support semi-sync replication, where at least two machines persist the writes before considering the write successful. Since the replicas were not ready, this cluster was not usable by Convex. Human operators were paged to intervene and fix the broken automation. We're working with our database partners to better understand what went wrong and what changes will prevent it in the future.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-04-09 05:42 UTC | Major incident: Small percentage of deployments experiencing outage (this change) | outage |

All Convex changes: [timeline](https://rewindex.com/company/convex/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i6113ded184e058 · Markdown: https://rewindex.com/change/i6113ded184e058.md · JSON: https://rewindex.com/api/v1/changes/i6113ded184e058 · Generated 2026-10-18T02:28:18.502Z