ChangesOrbElevated errors across API endpoints
Orb: Major incident: Elevated errors across API endpoints
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Orb removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Elevated errors across API endpoints
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-09-03 13:28 UTC
Before: not present
After: 2025-09-03 14:36 UTC
Before: not present
After: Analytics, API - Analytics, API - Reads, API - Writes, Ingest, Reads, Writes
Before: not present
After: We are currently investigating database issues causing API errors
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We're continuing to investigate, and are actively pursuing mitigation strategies. We apologize for the disruption and will provide status updates diligently here as we learn more.
Before: not present
After: We are continuing to see elevated errors on write endpoints. We believe we understand the root cause, and are pursuing multiple parallel mitigation strategies to resolve the incident as quickly as possible.
Before: not present
After: Although this incident is still active, we're seeing partial recovery for specific customers. We're continuing to treat this as top priority and working to mitigate the impact by running maintenance operations at our database layer.
Before: not present
After: We're seeing persistent partial recovery across writes, but a lower rate of failures persist. We believe the mitigations we've put in place are helping, but are continuing to pursue faster and more encompassing mitigation strategies.
Note that the ingestion API is not failing and has not during the incident; once the incident is resolved we do not expect any data gaps with event ingestion so no retries should be necessary.
Before: not present
After: We're continuing to focus on mitigating impact. Once again, we apologize for the disruption and will publish an RCA after the incident is resolved.
Before: not present
After: We are continuing to investigate the issue.
Before: not present
After: We're seeing broader recovery, and have seen no recent API errors since 12:38 UTC and now working to bring back async services.
Before: not present
After: Continuing to see broader recovery - API errors have recovered.
Before: not present
After: We're continuing to see broad recovery, and there have been no API errors since 12:38 UTC. Continuing to work to bring back async services.
Before: not present
After: Our services are continuing to recover, including issuing any outstanding invoices and webhooks. We have not seen any elevated rate of API errors since initial recovery.
Before: not present
After: We are continuing to work through our asynchronous work queue.
Before: not present
After: We are continuing to work through our asynchronous work queue.
Before: not present
After: We are nearly recovered, but will keep the incident in monitoring state until all asynchronous work is fully stable.
Before: not present
After: We are continuing to work through our asynchronous work queue.
Before: not present
After: We are nearing full resolution, and will continue to keep this incident updated.
Before: not present
After: This incident has been resolved. We'll continue to monitor for any disruptions, and follow up with a detailed RCA.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Wednesday, September 3, 2025
1 incident change
- MajorMajor incident
Elevated errors across API endpointsthis change