Companies / AssemblyAI / incidents

AssemblyAI: Async + LLM gateway APIs returning 5xx

Affected components: DashboardWeb - DashboardAPIs - Streaming APIStreaming APIWebsiteWeb - WebsiteAPIs - Asynchronous APIAsynchronous APIAPIs - LLM GatewayLLM GatewayPlaygroundWeb - Playground

Timeline of updates

1. investigating2026-01-29 17:54 UTC

   We are currently investigating an issue affecting async + LLM gateway services. More updates to come as we learn more.

2. investigating2026-01-29 18:02 UTC

   We are continuing to investigate this issue.

3. investigating2026-01-29 18:09 UTC

   We are currently investigating an issue affecting our Async and LLM Gateway services, which are returning 500 errors. Our team is actively working to identify and resolve the problem. We will provide updates as more information becomes available. All streaming API and EU endpoint services are running normally. This incident is isolated to NA API endpoints for async + LLM gateway.

4. investigating2026-01-29 18:29 UTC

   We are seeing requests beginning to successfully complete and have moved to the monitoring phase. Some requests are still returning 500 errors. Our team continues to actively investigate and monitor the situation. We will provide updates as more information becomes available.

5. monitoring2026-01-29 18:30 UTC

   We are seeing requests beginning to successfully complete and have moved to the monitoring phase. Some requests are still returning 500 errors. Our website is down, and logins may be affected. Our team continues to actively investigate and monitor the situation. We will provide updates as more information becomes available.

6. monitoring2026-01-29 18:42 UTC

   We are continuing to see requests being completed. We are working through a backlog of requests, resulting in elevated turnaround times while we recover. Our team continues to actively monitor the situation. We will provide updates as more information becomes available.

7. monitoring2026-01-29 19:18 UTC

   We are continuing to work through the backlog, and turnaround times are improving. Our team continues to actively monitor the situation. We will provide updates as more information becomes available.

8. resolved2026-01-29 19:32 UTC

   We have cleared the backlog, and turnaround times are back to normal. Our team continues to actively monitor the situation. We will provide a post-mortem when available.

9. postmortem2026-02-05 18:35 UTC

   **Summary: January 29 Outage** On January 29, our Async Transcription and LLM Gateway went down for 25 minutes, followed by two hours of degraded performance. Here's what happened. Our traffic has been steadily increasing and we received an influx of batch transcription jobs that were higher than usual. Autoscaling did what it should—spun up workers. We ran out of spot instances, fell back to on-demand, then hit AWS's Elastic Network Interface limit. Every ECS task needs an ENI. With no ENIs available, our authentication service couldn't rotate its tasks. It scaled to zero. Everything stopped. When we tried to recover, a Lambda function meant to help with failures was hammering the AWS API with describe calls, rate-limiting the very operations we needed to bring things back. A tool built for recovery was blocking recovery. **What actually went wrong:** We didn't monitor ENI utilization and our authentication service shared resources with batch workers. Traffic spike in one starved the other. **What we're doing:** Short-term: ENI alerts, isolate auth on dedicated capacity, fix the Lambda. Medium-term: Full multi-region for everything. This was preventable. We had the warning signs and didn't connect them. We're sorry, and we're fixing it.

Rewindex changes for this incident

• Critical incident: Async + LLM gateway APIs returning 5xx 2026-01-29 17:54 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). AssemblyAI profile · Recent AssemblyAI incidents · Markdown · JSON