Changes / Convex / Small percentage of deployments experiencing outage
Convex: Major incident: Small percentage of deployments experiencing outage
Summary
A small percentage of Convex deployments experienced sporadic availability in a major-impact outage from 2026-04-09 05:42 UTC to 2026-04-09 06:24 UTC (42 min). During database primary failover, orchestration failed to provision a replica ready for new writes; Convex implemented a fix and reported resolution.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Small percentage of deployments experiencing outage
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-04-09 05:42 UTC
Before: not present
After: 2026-04-09 06:24 UTC
Before: not present
After: Business, Development Services - Business, Development Services - Free & Starter, Development Services - Professional, Free & Starter, Live Traffic - Business, Live Traffic - Free & Starter, Live Traffic - Professional, Professional
Before: not present
After: Seems related to a problematic database failover. Updates soon.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This issue is considered resolved.
Impact: A small percentage of Convex deployments experienced sporadic availability for approximately 17 minutes, from 22:42 to 22:59 PM PT.
Root cause: one of our database clusters experienced a routine primary failover event. During the primary promotion process, the orchestration system failed to provision a replica that was ready to accept new database writes.
Out of an abundance of caution, Convex will only use database clusters that support semi-sync replication, where at least two machines persist the writes before considering the write successful.
Since the replicas were not ready, this cluster was not usable by Convex. Human operators were paged to intervene and fix the broken automation. We're working with our database partners to better understand what went wrong and what changes will prevent it in the future.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Small percentage of deployments experiencing outage (this change) A small percentage of Convex deployments experienced sporadic availability in a major-impact outage from 2026-04-09 05:42 UTC to 2026-04-09 06:24 UTC (42 min). During database primary failover, orchestration failed to provision a replica ready for new writes; Convex implemented a fix and reported resolution. | Major incident |