rewindex

ChangesLiveKitIncreased join latency in US East and US Central

LiveKit: Incident updated: Increased join latency in US East and US Central

Incident updateNormal progress (rule, 100%)Changed stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what LiveKit removed; blue is what they added. Muted text is unchanged context.

ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Root Cause Between 2026-10-16 and 2026-10-23, some new connections to LiveKit Cloud in US Central, US East and EU Central got no response. The cause was a fault in our cloud provider's network load balancers. Some load balancers kept sending new connections to servers that were no longer in service. Most clients connected on a retry, so the main effect was a slower join. Some clients waited much longer. A routine server replacement and one of our recovery actions on October 20 also closed open connections from self-hosted agent workers. This caused agent dispatch failures for some customers. Technical Details Each LiveKit Cloud region has network load balancers from our cloud provider. A load balancer sends new connections only to servers that pass its health check. When servers are added or removed, the provider updates the health-check configuration of the load balancer. Our cloud provider found these causes: A burst of health-check configuration updates overloaded the provider's health-check service. Some of these updates failed. A bug in the automatic recovery left the affected health checks inactive after the overload stopped. As a result, the load balancers kept old health data. They continued to send new connections to servers that were no longer listening. Those connections got no response. The fault affected only some load balancers. Other load balancers with the same servers worked correctly. The load balancers reported no errors. Agent dispatch failures. Each self-hosted agent worker keeps one long-lived connection to LiveKit Cloud. The worker gets job offers on this connection. Two events on October 20 closed these connections: At 16:14, a routine server replacement in US Central stopped one of our ingress servers. This closed about 485 agent worker connections at the same time. At that time, new traffic did not go to US Central, so these workers reconnected to US East. The workers of each affected customer were then split between two regions. Until about 17:40, agent dispatch failed or was slow for some customers. At 23:15, we deleted the faulty load balancer in US Central. About 600 agent worker connections still used it. The workers reconnected within about 1 minute. But our servers did not detect that the old connections were closed. They continued to offer jobs on these connections until 23:32, when the connections timed out. Those job offers failed, and some jobs were not dispatched. Timeline (UTC) 2026-10-16 06:50 - Connection timeouts started in EU Central 2026-10-16 10:52 - We moved traffic away from the affected EU Central servers; the timeouts stopped. We opened a case with our cloud provider 2026-10-17 - We added probes that test each load balancer address from several regions 2026-10-20 08:00 - Slow agent joins started in US Central 2026-10-20 15:28 - We moved traffic away from US Central; join times went back to normal 2026-10-20 16:14 - A routine server replacement in US Central closed about 485 agent worker connections; agent dispatch failed or was slow for some customers until about 17:40 2026-10-20 21:30 - After traffic returned to US Central, slow joins started in US Central and US East 2026-10-20 22:30 - We moved traffic away from US Central again 2026-10-20 23:13 - We moved traffic away from US East 2026-10-20 23:15 - We deleted the faulty load balancer in US Central; about 600 agent worker connections still used it 2026-10-20 23:32 - The old agent worker connections timed out; agent dispatch went back to normal 2026-10-20 23:41 - We replaced the faulty load balancer in US Central and moved traffic back 2026-10-21 00:55 - We replaced the faulty load balancers in US East and moved traffic back 2026-10-21 03:13 - The fault occurred on another load balancer in US Central; we moved traffic away 2026-10-21 06:12 - We replaced that load balancer and moved traffic back to US Central 2026-10-21 07:02 - We replaced the faulty load balancers in EU Central and moved traffic back 2026-10-21 - Our cloud provider restored the correct health state on the 5 affected load balancers 2026-10-23 15:15 - The fault occurred at a lower rate on one load balancer in US East 2026-10-23 22:35 - Our probes detected it 2026-10-23 23:10 - We moved traffic off that load balancer; customer impact ended 2026-10-27 - Our cloud provider sent the root cause analysis Scope of Impact On each affected load balancer, 5% to 25% of new connections got no response. The load balancer fault did not affect connections that were already open. But a routine server replacement and one of our recovery actions on October 20 closed open connections from self-hosted agent workers (see Technical Details). Time (UTC) Region Effect Oct 16 06:50–10:52 EU Central About 5% of new connections timed out. This included connections for SIP calls. Oct 20 08:00 – Oct 21 06:15 US Central A share of new connections timed out. Oct 20 16:14–17:40 US Central, US East Some self-hosted agents did not get jobs, or joined late. Some inbound calls were not answered. Oct 20 21:30 – Oct 21 00:55 US East 5% to 10% of new connections timed out. Oct 20 23:16–23:32 US Central Job offers to some self-hosted agent workers timed out. Some jobs were not dispatched. Oct 23 15:15–18:15 and 20:45–23:10 US East About 0.3–0.5% of new connections to the region timed out. The LiveKit SDKs stop a signal connection attempt after 5 seconds and try again. Most participants and agents joined on the retry, about 5 seconds later than usual. Clients with a longer connection timeout, or with no timeout, waited longer. Some self-hosted agents reported room joins that took 60–70 seconds. API calls to the affected regions could also time out or be slow. Detection and Response Customers reported the EU Central and US Central timeouts before our monitoring found them. Our checks tested each region as a whole, so a fault on one load balancer address was hidden. Most connections in a region still worked. On October 17, we added probes that test each load balancer address from several regions. These probes detected the October 23 fault. In each event, we moved traffic away from the affected region, replaced the faulty load balancers, and then moved traffic back. Mitigations and Follow-ups Our cloud provider fixed the bug in a new release. With the fix, a failed health-check update is retried until it succeeds. The provider plans to start the deployment of this release in mid-November. We are working with the provider on when the fix reaches each region. Each load balancer now sends connections only to the servers that run our ingress servers. Before, it included a larger group of servers, so each server that we added or removed caused a health-check update. Now the health checks change only when an ingress server moves. This change is live in most regions. We are completing it in the remaining regions, including US Central and US East. Our probes now test each load balancer address and alert when the same connections fail again and again. We changed how we replace a faulty load balancer. We now add a new load balancer, test it, and move traffic to it. The faulty load balancer stays in service until its open connections end, so agent workers on it are not cut off. If an agent dispatch times out in one region, LiveKit Cloud now tries the next region. We are working on a change that moves agent worker connections out of a region when we move traffic away from it. Then all agent workers of a customer reconnect to the same region. We sincerely apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdayThursday, November 5Thu, Nov 5

1 rewrite, 1 incident change

Wednesday, October 21

3 rewrites

Tuesday, October 20

2 incident changes

Cite this

Incident updated: Increased join latency in US East and US Central on LiveKit’s status page, changed 2026-11-05 22:34 UTC (change 6 of 7).

Title
Incident updated: Increased join latency in US East and US Central
Vendor
LiveKit
Date
URL
https://rewindex.com/change/r73fd331bdf1f9d

All LiveKit changesLiveKit profileMarkdownJSON