Retell AI: Web Calls and Phone Calls Disruption
Affected components: End to End Calling - Phone CallPhone CallEnd to End Calling - Web CallWeb Call
Timeline of updates
- investigating
Starting at 4:10 PM, web and outbound calls began failing, while inbound calls experienced connection delays. We are actively investigating the issue and will provide an update as soon as more information is available.
- resolved
The issue affecting web, outbound, and inbound calls has been resolved. All calling services are operating normally.
- postmortem
### Summary On September 5, a rare race condition in our custom LLM call path caused call scheduling queues across our fleet to stall. Because the failure also made containers appear underutilized, autoscaling never triggered. Between roughly 10:00 and 16:00 PST the impact grew until most containers were stuck and new calls could not be placed. ### Impact * **Web calls, outbound calls, and inbound calls** were all affected. * Inbound calls were the least affected, since they hold the highest priority in our scheduling queue. * Peak impact occurred around 16:00 PST. ### Root Cause The failure required a specific sequence of events: 1. **Stalled initialization.** If a call ended before its custom LLM websocket was established, the call's initialization never completed. The scheduling queue waited indefinitely for that initialization to finish. 2. **Queue exhaustion.** Each container allows a fixed number of concurrent dequeue operations. Enough stalled initializations consumed all available slots, leaving the container unable to pick up new calls. 3. **Misleading scaling signals.** The call-end path still ran and decremented the container's concurrency counter. As a result, stuck containers reported low concurrency and low resource utilization — so no scale-out was triggered. With every container blocked and autoscaling idle, new calls had nowhere to go. This pattern is only reachable through that exact ordering, which is why it went undetected until it occurred at scale. ### Timeline \(PST\) * 10AM: A customer's custom LLM calls begin dropping before initialization completes. * Afternoon: The customer scales up traffic; the number of stalled initializations grows. * ~16:00: Nearly all containers are blocked on dequeue slots. Autoscaling does not trigger. Call placement is broadly impacted. * After 16:00: Mitigation begins: manual scale-out of containers and clearing of stuck concurrency for the affected customer. Service recovers. ### Prevention 1. \(COMPLETED\): Fix the race condition in custom LLM call initialization 2. \(IN PROGRESS\): Add monitoring and alerting on call scheduling queues to detect stuck states 3. \(IN PROGRESS\): Decouple the scheduling queue from call initialization so a slow or failed init cannot block dequeue 4. \(IN PROGRESS\): Add call scheduling queue pressure as an input to our autoscaling policies ### Closing We apologize for the disruption. We are treating this as an opportunity to harden our scheduling and scaling infrastructure, and we will ship the remaining action items within a few days.
Changes Rewindex caught on this incident
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).