# Orb: Major incident: Elevated errors across API endpoints

**Summary:** _summary pending_
- Kind: Major incident · Label: major (vendor)
- Changed: 2025-09-03 13:28 UTC · Detected: 2026-10-19 18:53 UTC
- [Source: https://stspg.io/dxt0mkq9rkjy](https://stspg.io/dxt0mkq9rkjy)
- [Company: Orb](https://rewindex.com/company/orb.md)
- Versions: before (none: first archived) → after incident\_version:752/58fvl5xf2sfz@913e3403e635
- Change 1 of 1 for this incident

Incident page (full timeline): [Elevated errors across API endpoints](https://rewindex.com/company/orb/incidents/58fvl5xf2sfz.md)
## What changed

**1. Added: Title**

```diff
+ Elevated errors across API endpoints
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2025-09-03 13:28 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-09-03 14:36 UTC
```

**6. Added: Affected components**

```diff
+ Analytics, API - Analytics, API - Reads, API - Writes, Ingest, Reads, Writes
```

**7. Added: Update** (investigating) · 2025-09-03 10:46 UTC

```diff
+ We are currently investigating database issues causing API errors
```

**8. Added: Update** (investigating) · 2025-09-03 10:51 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (investigating) · 2025-09-03 11:14 UTC

```diff
+ We're continuing to investigate, and are actively pursuing mitigation strategies. We apologize for the disruption and will provide status updates diligently here as we learn more.
```

**10. Added: Update** (investigating) · 2025-09-03 11:31 UTC

```diff
+ We are continuing to see elevated errors on write endpoints. We believe we understand the root cause, and are pursuing multiple parallel mitigation strategies to resolve the incident as quickly as possible.
```

**11. Added: Update** (investigating) · 2025-09-03 11:45 UTC

```diff
+ Although this incident is still active, we're seeing partial recovery for specific customers. We're continuing to treat this as top priority and working to mitigate the impact by running maintenance operations at our database layer.
```

**12. Added: Update** (investigating) · 2025-09-03 11:57 UTC

```diff
+ We're seeing persistent partial recovery across writes, but a lower rate of failures persist. We believe the mitigations we've put in place are helping, but are continuing to pursue faster and more encompassing mitigation strategies.  Note that the ingestion API is not failing and has not during the incident; once the incident is resolved we do not expect any data gaps with event ingestion so no retries should be necessary.
```

**13. Added: Update** (investigating) · 2025-09-03 12:15 UTC

```diff
+ We're continuing to focus on mitigating impact. Once again, we apologize for the disruption and will publish an RCA after the incident is resolved.
```

**14. Added: Update** (investigating) · 2025-09-03 12:32 UTC

```diff
+ We are continuing to investigate the issue.
```

**15. Added: Update** (monitoring) · 2025-09-03 12:45 UTC

```diff
+ We're seeing broader recovery, and have seen no recent API errors since 12:38 UTC and now working to bring back async services.
```

**16. Added: Update** (monitoring) · 2025-09-03 12:46 UTC

```diff
+ Continuing to see broader recovery - API errors have recovered.
```

**17. Added: Update** (monitoring) · 2025-09-03 13:03 UTC

```diff
+ We're continuing to see broad recovery, and there have been no API errors since 12:38 UTC. Continuing to work to bring back async services.
```

**18. Added: Update** (monitoring) · 2025-09-03 13:17 UTC

```diff
+ Our services are continuing to recover, including issuing any outstanding invoices and webhooks. We have not seen any elevated rate of API errors since initial recovery.
```

**19. Added: Update** (monitoring) · 2025-09-03 13:28 UTC

```diff
+ We are continuing to work through our asynchronous work queue.
```

**20. Added: Update** (monitoring) · 2025-09-03 13:47 UTC

```diff
+ We are continuing to work through our asynchronous work queue.
```

**21. Added: Update** (monitoring) · 2025-09-03 14:00 UTC

```diff
+ We are nearly recovered, but will keep the incident in monitoring state until all asynchronous work is fully stable.
```

**22. Added: Update** (monitoring) · 2025-09-03 14:18 UTC

```diff
+ We are continuing to work through our asynchronous work queue.
```

**23. Added: Update** (monitoring) · 2025-09-03 14:32 UTC

```diff
+ We are nearing full resolution, and will continue to keep this incident updated.
```

**24. Added: Update** (resolved) · 2025-09-03 14:36 UTC

```diff
+ This incident has been resolved. We'll continue to monitor for any disruptions, and follow up with a detailed RCA.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-09-03 13:28 UTC | Major incident: Elevated errors across API endpoints (this change) | major |

All Orb changes: [timeline](https://rewindex.com/company/orb/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ia9b4efc75c29bc · Markdown: https://rewindex.com/change/ia9b4efc75c29bc.md · JSON: https://rewindex.com/api/v1/changes/ia9b4efc75c29bc · Generated 2026-10-20T00:48:41.976Z