ChangesRetell AICall service disruption
Retell AI: Major incident: Call service disruption
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Retell AI removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Call service disruption
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-09-17 01:35 UTC
Before: not present
After: 2026-09-17 01:35 UTC
Before: not present
After: End to End Calling - Phone Call, End to End Calling - Web Call, Phone Call, Web Call
Before: not present
After: We are aware of issues with some calls and are investigating.
Before: not present
After: The issue has been found and resolved. About 22% of calls may have been affected and been ended early, or have no audio.
A postmortem will be published by EOD 09-17.
Before: not present
After: **Status: resolved.** All times are PST.
## Summary
On September 16 at 5:37 PM we deployed a planned improvement to call handling. The improvement was not meant to be active yet, but the deployment turned it on, and it began to assign live calls to workers.
This change unintentionally let two workers take the same call, and each worker then started it. The two starts interfered with each other, and the call failed. About 22% of calls were affected from 5:37 PM to 6:20 PM.
We stopped the new workers, rolled the deployment back, and saw the issue resolve.
## Impact
An affected call shows one of two signs:
* The call ends as soon as it connects. The duration is zero or close to zero seconds.
* The call connects. The agent does not speak. The call ends after a silence.
A call that ran correctly is not affected. Its recording, its transcript, and its billing record are correct.
## Root cause
We were making an improvement to our system for call handling. As part of this change, we made a new queue for calls and a new set of workers, which was meant to be inactive. Our old servers were writing to both queues as a test, which did not cause issues as the new workers were not active.
A deployment replaces servers one group at a time, so when the deployment began at 5:37PM, some servers ran the old code and some ran the new code. The old servers still wrote each call to both queues, and the new workers now read the new queue. The two systems therefore handled the same call, once each, and undid each other's work.
## Timeline \(PST\)
| Time | Event |
| --- | --- |
| 5:37 PM | We deploy the change. |
| 5:40 PM | Call errors increase. |
| 5:43 PM | Our call monitor reports the first failures. |
| 5:47 PM | The system pages the on-call engineer. |
| 6:14 PM | We stop the new call workers. Double handling drops sharply. |
| 6:18 PM | We start the rollback, which removes the change itself. |
| 6:20 PM | The last affected call. Call errors return to normal. |
| 7:54 PM | The rollback finishes. |
## Prevention
1. \(IN PROGRESS\) **Improve rollout process checks** We will improve our process checks for large changes to ensure that future deployments roll out in the intended state, and explicitly call out deployment time checks.
2. \(IN PROGRESS\) **Let the monitor stop a deployment.** We will connect our call monitor to our deployment system. Repeated failures will then stop a deployment.
## Closing
We are sorry for the disruption. A failed call is a bad experience for you, and for the person you called.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Thursday, September 17
1 incident change
- MajorMajor incident
Call service disruptionthis change