Changes / Convex / elevated latency/errors for a subset of customers
Convex: Major incident: elevated latency/errors for a subset of customers
Summary
Convex reported major latency/errors for affected deployments starting 2025-10-30 23:43 UTC, resolved 2025-10-31 07:52 UTC after 8 h 8 min. A downstream-provider issue caused an inefficient database table format; Convex reported rewriting it and shedding load, with sporadic availability issues for a small subset of non-pro customers.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 70%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: elevated latency/errors for a subset of customers
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-10-30 23:43 UTC
Before: not present
After: 2025-10-31 07:52 UTC
Before: not present
After: Free & Starter, Live Traffic - Database Services, Live Traffic - Function Runtime
Before: not present
After: We're investigating an issue with a database cluster leading to degraded performance and higher error rates for affected deployments.
Before: not present
After: We've restored affected pro deployments. Working on bringing back affected free deployments.
Before: not present
After: we've brought back a subset of free deployments. Continuing to remediate.
Before: not present
After: We're continuing to work on a fix for the issue. Expect another update in ~90 minutes.
Before: not present
After: All deployments should be restored. We are monitoring the database to ensure it remains stable.
Before: not present
After: There will be sporadic availability issues for a small subset of non-pro customers. We apologize for this impact. There was an issue with a downstream provider which resulted in a database cluster running with an inefficient table format. We're in the process of rewriting this data format but have had to shed load to provide headroom while reoptimizing the database.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: elevated latency/errors for a subset of customers (this change) Convex reported major latency/errors for affected deployments starting 2025-10-30 23:43 UTC, resolved 2025-10-31 07:52 UTC after 8 h 8 min. A downstream-provider issue caused an inefficient database table format; Convex reported rewriting it and shedding load, with sporadic availability issues for a small subset of non-pro customers. | Major incident |