rewindex

ChangesLiveKitInvestigating failing calls and RoomService API failures in US Central/East and dashboard issues

LiveKit: Incident updated: Investigating failing calls and RoomService API failures in US Central/East and dashboard issues

Incident updateNot classified yetChanged status.livekit.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what LiveKit removed; blue is what they added. Muted text is unchanged context.

ChangedStatus#1
Before: monitoring
After: resolved
AddedResolved time#2
Before: not present
After: 2026-10-29 01:19 UTC
AddedUpdateresolved#3
Before: not present
After: This incident has been resolved. Between 23:30 and 23:58 UTC, a network disruption at an upstream infrastructure provider affected our US East region, with impact also seen in US Central. During this window: Inbound SIP calls in US East and US Central failed at elevated rates, with affected calls rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55 UTC. About 10% of outbound SIP call requests (CreateSIPParticipant) in US East failed, some returning 408 deadline_exceeded. Between 23:28 and 23:58 UTC, there were elevated rates of Room Service, Egress, and Ingress API request server errors in US East. Some participants in US East and US Central saw failed room joins or increased packet loss. The LiveKit Cloud dashboard was inaccessible for part of this window. Network connectivity recovered at approximately 23:58 UTC, and error rates and call volumes returned to normal by 00:20 UTC. A related network issue also affected our much smaller Mexico region from 23:29 UTC, disconnecting active sessions and degrading service until traffic was moved to nearby regions at approximately 01:01 UTC. Sincere apologies for the impact. We will follow up with a postmortem.

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdaySunday, November 1Sun, Nov 1

1 rewrite, 2 incident changes

Thursday, October 29

4 incident changes

All LiveKit changesLiveKit profileMarkdownJSON