# AssemblyAI: Critical incident: Async + LLM gateway APIs returning 5xx

**Summary:** AssemblyAI’s Async Transcription and LLM Gateway APIs returned 5xx errors in a critical incident from 2026-01-29 17:54 UTC to 2026-01-29 19:32 UTC (1 h 38 min). Recovery involved elevated turnaround times while requests backlogged; AssemblyAI reported clearing the backlog and restoring normal turnaround times. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-01-29 17:54 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/crrrqclrylss](https://stspg.io/crrrqclrylss)
- [Company: AssemblyAI](https://rewindex.com/company/assemblyai.md)
- Versions: before (none: first archived) → after incident\_version:2/cshs4z0mkvr7@a631f03590d9
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Async + LLM gateway APIs returning 5xx
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2026-01-29 17:54 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-01-29 19:32 UTC
```

**6. Added: Affected components**

```diff
+ APIs - Asynchronous API, APIs - LLM Gateway, APIs - Streaming API, Asynchronous API, Dashboard, LLM Gateway, Playground, Streaming API, Web - Dashboard, Web - Playground, Website, Web - Website
```

**7. Added: Update** (investigating) · 2026-01-29 17:54 UTC

```diff
+ We are currently investigating an issue affecting async + LLM gateway services. More updates to come as we learn more.
```

**8. Added: Update** (investigating) · 2026-01-29 18:02 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (investigating) · 2026-01-29 18:09 UTC

```diff
+ We are currently investigating an issue affecting our Async and LLM Gateway services, which are returning 500 errors. Our team is actively working to identify and resolve the problem. We will provide updates as more information becomes available. All streaming API and EU endpoint services are running normally. This incident is isolated to NA API endpoints for async + LLM gateway.
```

**10. Added: Update** (investigating) · 2026-01-29 18:29 UTC

```diff
+ We are seeing requests beginning to successfully complete and have moved to the monitoring phase. Some requests are still returning 500 errors. Our team continues to actively investigate and monitor the situation. We will provide updates as more information becomes available.
```

**11. Added: Update** (monitoring) · 2026-01-29 18:30 UTC

```diff
+ We are seeing requests beginning to successfully complete and have moved to the monitoring phase. Some requests are still returning 500 errors. Our website is down, and logins may be affected. Our team continues to actively investigate and monitor the situation. We will provide updates as more information becomes available.
```

**12. Added: Update** (monitoring) · 2026-01-29 18:42 UTC

```diff
+ We are continuing to see requests being completed. We are working through a backlog of requests, resulting in elevated turnaround times while we recover. Our team continues to actively monitor the situation. We will provide updates as more information becomes available.
```

**13. Added: Update** (monitoring) · 2026-01-29 19:18 UTC

```diff
+ We are continuing to work through the backlog, and turnaround times are improving. Our team continues to actively monitor the situation. We will provide updates as more information becomes available.
```

**14. Added: Update** (resolved) · 2026-01-29 19:32 UTC

```diff
+ We have cleared the backlog, and turnaround times are back to normal. Our team continues to actively monitor the situation. We will provide a post-mortem when available.
```

**15. Added: Update** (postmortem) · 2026-02-05 18:35 UTC

```diff
+ **Summary: January 29 Outage** On January 29, our Async Transcription and LLM Gateway went down for 25 minutes, followed by two hours of degraded performance. Here's what happened. Our traffic has been steadily increasing and we received an influx of batch transcription jobs that were higher than usual. Autoscaling did what it should—spun up workers. We ran out of spot instances, fell back to on-demand, then hit AWS's Elastic Network Interface limit. Every ECS task needs an ENI. With no ENIs available, our authentication service couldn't rotate its tasks. It scaled to zero. Everything stopped. When we tried to recover, a Lambda function meant to help with failures was hammering the AWS API with describe calls, rate-limiting the very operations we needed to bring things back. A tool built for recovery was blocking recovery. **What actually went wrong:** We didn't monitor ENI utilization and our authentication service shared resources with batch workers. Traffic spike in one starved the other. **What we're doing:** Short-term: ENI alerts, isolate auth on dedicated capacity, fix the Lambda. Medium-term: Full multi-region for everything. This was preventable. We had the warning signs and didn't connect them. We're sorry, and we're fixing it.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-01-29 17:54 UTC | Critical incident: Async + LLM gateway APIs returning 5xx (this change) | outage |

All AssemblyAI changes: [timeline](https://rewindex.com/company/assemblyai/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ieed4cd9e2a4680 · Markdown: https://rewindex.com/change/ieed4cd9e2a4680.md · JSON: https://rewindex.com/api/v1/changes/ieed4cd9e2a4680 · Generated 2026-10-18T01:30:17.060Z