Changes / Convex / Convex traffic having downtime
Convex: Major incident: Convex traffic having downtime
Summary
Convex reported a major incident causing intermittent unavailability across all Convex services, starting 2025-09-20 12:18 UTC and resolved at 2025-09-20 13:15 UTC (57 min). The postmortem cited a cascading traffic-layer failure after a Caddy node ran out of memory during an unforeseen load spike; Convex added capacity.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Convex traffic having downtime
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-09-20 12:18 UTC
Before: not present
After: 2025-09-20 13:15 UTC
Before: not present
After: We are currently investigating this issue.
Before: not present
After: An unexpected traffic pattern overloaded some of our services, causing intermittent unavailability across Convex instances. We've added extra capacity and are monitoring to ensure that the system is stable.
Before: not present
After: This incident has been resolved.
Before: not present
After: From around 5:18am to 5:54am Pacific \(12:18pm to 12:54pm UTC\), Convex had a 36 min period of intermittent downtime that affected all Convex services.
The specific issue was a cascading failure in our traffic layer. We had a traffic node \(Caddy\) run out of memory due to an unforeseen load spike and instead of just being restarted/replaced this node was marked as permanently down by our container management layer \(Nomad\) which led to the issue propagating to all traffic servers.
Since the incident we've more than doubled the size of our traffic layer, fixed the failover behavior which led to nodes staying failed after OOMing, and will be investigating alternative traffic services.
As always data was safe during this incident but we really apologize for the availability impact during that time period.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Convex traffic having downtime (this change) Convex reported a major incident causing intermittent unavailability across all Convex services, starting 2025-09-20 12:18 UTC and resolved at 2025-09-20 13:15 UTC (57 min). The postmortem cited a cascading traffic-layer failure after a Caddy node ran out of memory during an unforeseen load spike; Convex added capacity. | Major incident |