Changes / AssemblyAI / Async + LLM gateway APIs returning 5xx
AssemblyAI: Critical incident: Async + LLM gateway APIs returning 5xx
Summary
AssemblyAI’s Async Transcription and LLM Gateway APIs returned 5xx errors in a critical incident from 2026-01-29 17:54 UTC to 2026-01-29 19:32 UTC (1 h 38 min). Recovery involved elevated turnaround times while requests backlogged; AssemblyAI reported clearing the backlog and restoring normal turnaround times.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Async + LLM gateway APIs returning 5xx
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2026-01-29 17:54 UTC
Before: not present
After: 2026-01-29 19:32 UTC
Before: not present
After: APIs - Asynchronous API, APIs - LLM Gateway, APIs - Streaming API, Asynchronous API, Dashboard, LLM Gateway, Playground, Streaming API, Web - Dashboard, Web - Playground, Website, Web - Website
Before: not present
After: We are currently investigating an issue affecting async + LLM gateway services. More updates to come as we learn more.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We are currently investigating an issue affecting our Async and LLM Gateway services, which are returning 500 errors. Our team is actively working to identify and resolve the problem. We will provide updates as more information becomes available.
All streaming API and EU endpoint services are running normally. This incident is isolated to NA API endpoints for async + LLM gateway.
Before: not present
After: We are seeing requests beginning to successfully complete and have moved to the monitoring phase. Some requests are still returning 500 errors. Our team continues to actively investigate and monitor the situation. We will provide updates as more information becomes available.
Before: not present
After: We are seeing requests beginning to successfully complete and have moved to the monitoring phase. Some requests are still returning 500 errors. Our website is down, and logins may be affected. Our team continues to actively investigate and monitor the situation. We will provide updates as more information becomes available.
Before: not present
After: We are continuing to see requests being completed. We are working through a backlog of requests, resulting in elevated turnaround times while we recover. Our team continues to actively monitor the situation. We will provide updates as more information becomes available.
Before: not present
After: We are continuing to work through the backlog, and turnaround times are improving. Our team continues to actively monitor the situation. We will provide updates as more information becomes available.
Before: not present
After: We have cleared the backlog, and turnaround times are back to normal. Our team continues to actively monitor the situation. We will provide a post-mortem when available.
Before: not present
After: **Summary: January 29 Outage**
On January 29, our Async Transcription and LLM Gateway went down for 25 minutes, followed by two hours of degraded performance. Here's what happened.
Our traffic has been steadily increasing and we received an influx of batch transcription jobs that were higher than usual. Autoscaling did what it should—spun up workers. We ran out of spot instances, fell back to on-demand, then hit AWS's Elastic Network Interface limit. Every ECS task needs an ENI. With no ENIs available, our authentication service couldn't rotate its tasks. It scaled to zero. Everything stopped.
When we tried to recover, a Lambda function meant to help with failures was hammering the AWS API with describe calls, rate-limiting the very operations we needed to bring things back. A tool built for recovery was blocking recovery.
**What actually went wrong:**
We didn't monitor ENI utilization and our authentication service shared resources with batch workers. Traffic spike in one starved the other.
**What we're doing:**
Short-term: ENI alerts, isolate auth on dedicated capacity, fix the Lambda.
Medium-term: Full multi-region for everything.
This was preventable. We had the warning signs and didn't connect them. We're sorry, and we're fixing it.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Async + LLM gateway APIs returning 5xx (this change) AssemblyAI’s Async Transcription and LLM Gateway APIs returned 5xx errors in a critical incident from 2026-01-29 17:54 UTC to 2026-01-29 19:32 UTC (1 h 38 min). Recovery involved elevated turnaround times while requests backlogged; AssemblyAI reported clearing the backlog and restoring normal turnaround times. | Major incident |
All AssemblyAI changes → · AssemblyAI profile · Markdown · JSON