rewindex

ChangesLiveKitInvestigating failing calls and RoomService API failures in US Central/East and dashboard issues

LiveKit: Incident updated: Investigating failing calls and RoomService API failures in US Central/East and dashboard issues

Incident updateNot classified yetChanged status.livekit.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what LiveKit removed; blue is what they added. Muted text is unchanged context.

ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. Impact US East (23:28 to 23:58 UTC) SIP inbound: a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55. SIP outbound: about 10% of CreateSIPParticipant requests in US East failed, some returning 408 deadline_exceeded. APIs: significant server errors on Room Service, Egress, and Ingress APIs in US East. Recent versions of our server SDKs automatically retry failed API requests in other regions, which reduced the impact for customers on those versions. Cloud Agents: agents in US East couldn't reach some of the model providers and other services they depend on, so agent sessions failed or degraded. Cloud Agents currently run in a single region, so they couldn't be quickly moved. Realtime: some participants in US East saw failed room joins or increased packet loss. Sessions that were already connected kept working. Dashboard: the LiveKit Cloud dashboard was inaccessible for part of the window. Mexico (23:29 to 01:01 UTC) Active sessions were disconnected and service was degraded until we moved traffic to nearby regions. Root Cause Starting at around 23:28 UTC, our cloud provider had a networking incident in its US East data center. It was a partial outage: our servers could still reach end users and our other regions, but lost access to our database and to certain external services, such as model providers. Anything that depended on those started failing. API requests and SIP call setup timed out, and agents couldn't reach the model providers they need to respond. Our automatic failover moves traffic away from a region when it becomes unhealthy. It didn't kick in here, for two reasons: Our health checks mostly look at our own software. None of it failed. Every server in US East was up and responding. We could see that US East couldn't reach external services, but nothing acted on that automatically. Those signals paged engineers, but they didn't feed into failover. As far as failover was concerned, US East looked healthy, so we kept sending traffic to a region that couldn't do much of the work. Cloud Agents had a separate gap. They currently run only in the data center specified in its deployment (i.e. US East) and we did not have a process to automatically migrate them quickly in case of a data center failure. Connectivity came back at around 23:58 UTC, once the provider mitigated the issue on their side. Timeline (all times in UTC, starting October 28) 23:28: Requests from US East to our database and certain external services start timing out. API errors follow shortly after. 23:35: Alerts fire for API errors in US East and an incident is opened. 23:42: Incident call starts. We initially focus on the API timeouts. 23:45: We remove an affected server to rule out a single bad instance. Failures continue. Since SIP, APIs, and agents are all affected, we start treating it as a regional network problem rather than a software bug. 23:56: Incident escalated. We start draining US East. 23:58: The provider's fix restores connectivity in US East. Impact in US East ends. 00:36: We identify degraded SIP, realtime, and API service in our Mexico region. 01:01: Mexico traffic moved to nearby regions, and service there recovers. Corrective Actions & Prevention Fail over based on external connectivity. We're adding checks for whether a region can reach the services it depends on, and failover will act on them. If a region can't do its job, it should be drained automatically, even when every server in it looks healthy. Deploy Agents to multiple data centers. We'll run agents in multiple data centers to have sufficient redundancy. Faster failover in agent dispatch. If a server or an entire data center stops responding, agent dispatch will hand the job to another agent within seconds. Multi-cloud redundancy. We're adding capacity with more than one cloud provider, so an outage at one provider can't take down a region.

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdaySunday, November 1Sun, Nov 1

1 rewrite, 2 incident changes

Thursday, October 29

4 incident changes

All LiveKit changesLiveKit profileMarkdownJSON