Struck in red is what LiveKit removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Root Cause
On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses.
Technical Details
When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4.
Timeline (UTC)
2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began
2026-10-16 15:41 - Second affected SIP server started with an IPv6 address
2026-10-16 ~15:48 - Customers began reporting failures to our support team
2026-10-16 15:58 - SIP traffic began moving away from the final affected cluster due to automatic cluster draining
2026-10-16 15:59 - Last affected call; customer impact ended
2026-10-16 17:08 - Began rolling out the long term fix requiring the usage of IPv4
2026-10-16 ~21:28 - Fix rollout completed
2026-10-16 ~22:30 - SIP traffic restored to the affected cluster
Scope of Impact
Between 14:49 and 15:59 UTC on 2026-10-16, SIP calls handled by the two affected servers in US West failed. During that window, about 14% of outbound calls in US West were rejected at setup and about 5% of inbound calls were answered and then dropped without audio.
Detection and Response
Our customers noticed this before our monitoring did. Our alerts watch for error-level logs, but carrier rejections are logged as routine events since they happen during normal operation. We also weren't tracking call failure rates on individual servers so our on-call engineers were paged in manually about an hour after the first failure.
Mitigations and Follow-ups
We deployed a fix so SIP servers only use IPv4 addresses when determining their public address.
We'll add safeguards and automated tests to ensure a SIP server's call setup messages only contain IPv4 addresses.
We are additionally adding improvements in monitoring so that we can proactively detect and prevent these sorts of failures.
We apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.
LiveKit updated the incident title from “Degraded SIP Performance” to “Degraded SIP Performance in US West,” clarifying the affected region. The edit provides no additional impact, timing, cause or fix details.
Cite this
Incident updated: Degraded SIP Performance in US West on LiveKit’s status page, changed 2026-10-22 14:33 UTC (change 2 of 2).
Title
Incident updated: Degraded SIP Performance in US West