ChangesLiveKitDisconnections and failed connections in multiple regions
LiveKit: Incident updated: Disconnections and failed connections in multiple regions
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what LiveKit removed; blue is what they added. Muted text is unchanged context.
Before: resolved
After: postmortem
Before: not present
After: Summary
On October 31, 2026, between 07:10 and 07:43 UTC, an uncommon data pattern surfaced a latent software bug which caused media servers in several regions to restart repeatedly. A small percentage of participants were disconnected and the vast majority automatically reconnected. Impact was concentrated in North America. We stopped the trigger at 07:43 UTC and deployed a fix to all regions by 08:49 UTC.
We apologize for the disruption this caused.
Root Cause
Applications can open their own data channel and send raw data over it, outside of LiveKit's standard data message format. Media servers forward this raw data to other participants in the room, wrapping it with the sender's identity so that receivers know who sent it. When a room spans multiple regions, the media servers hosting it also forward this data to each other. That server-to-server step sent the raw data without the sender's identity, and the receiving server assumed the identity would be present, which caused it to crash. This gap in our server-to-server protocol had been present for some time and was not introduced by a recent deployment.
Starting at 07:10 UTC, one application sent raw data over its own data channel in a room that spanned multiple regions. Every server that received the forwarded message crashed at the same time, while the server the sender was connected to kept running and kept sending. Participants on the crashed servers automatically reconnected to other servers. Those servers then began hosting the room, received the next message, and crashed as well. This cycle repeated until we stopped the trigger at 07:43 UTC. In total, media servers in 12 of our data centers restarted, some of them up to 10 times.
Timeline (UTC)
2026-10-31 07:10 - An application begins sending raw data over its own data channel in a multi-region room. Media servers in US East and Canada restart within seconds.
07:13 - Our monitoring detects the restarts and an incident is opened automatically.
07:10 to 07:43 - Media servers in 12 data centers restart repeatedly as participants reconnect to new servers.
07:43 - We block the traffic triggering the bug. There are no further restarts after this point, and customer impact ends.
07:46 - A fix for the bug is ready.
08:04 - The fix begins rolling out, starting with a small set of regions.
08:49 - The fix is deployed to all regions.
Scope of Impact
Realtime sessions:
Across all regions, about 1.5% were disconnected more than once within five minutes.
In US East, US Central, US West, Canada and Mexico, about 6% were disconnected more than once within five minutes.
The median time from disconnect to rejoin was about 3 seconds.
Agents:
Agents were disconnected and reconnected the same way as other participants, with the same median reconnect time of about 3 seconds. An agent that reconnects keeps its conversation state, so most conversations continued after a short gap. Fewer than 1% of active agent sessions did not recover.
Ingress, Egress and SIP:
Most active WHIP ingress streams were interrupted, and reconnects typically took about 40 seconds. Some WHIP ingress update and delete requests also failed.
About 5% of active egress sessions were disconnected. Most reconnected within a few seconds.
About 1% of SIP participants were disconnected, and half of them rejoined in under a second. Some SIP call transfers also failed.
APIs:
About 1.3% of Room Service API requests failed during the window. ListParticipants and GetParticipant were the most affected.
How to tell if you were affected: participants in affected sessions saw a reconnecting state, or a brief loss of audio and video. In session analytics, an affected participant appears to leave and rejoin the same room with the same identity between 07:10 and 07:43 UTC.
Detection and Response
Our monitoring detected the server restarts about 3 minutes after they began, and an incident was opened automatically. Identifying the specific room and traffic triggering the restarts took about 30 minutes. Once we found it, we blocked the trigger at 07:43 UTC and the restarts stopped immediately. We deployed a fix to all regions by 08:49 UTC.
We did not post to our status page while the incident was in progress, and published a notice only after it was resolved. Our original post mistakenly sent notifications as we initially over-estimated the scope of impact.
Mitigations and Follow-ups
We fixed the bug. Media servers now handle forwarded data that arrives without the sender's identity instead of crashing. This fix was deployed to all regions by 08:49 UTC.
Raw data forwarded between media servers now includes the sender's identity, so receiving servers always know who sent it.
We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.
Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.
How the archive is checked, and what it cannot prove.
Before captured
In the 2026-10-31 checkpoint, timestamped by FreeTSA and DigiCert on . Record hash
a743a128a9ddccdc…After captured
In the 2026-11-03 checkpoint, timestamped by FreeTSA and DigiCert on . Record hash
d86f1a3961a90886…
History of this incident
YesterdayTuesday, November 3Tue, Nov 3
1 incident change
- Normal progressIncident
Incident updateDisconnections and failed connections in multiple regionsthis change
Cite this
Incident updated: Disconnections and failed connections in multiple regions on LiveKit’s status page, changed 2026-11-03 14:03 UTC (change 1 of 1).
- Title
- Incident updated: Disconnections and failed connections in multiple regions
- Vendor
- LiveKit
- Date
- URL
- https://rewindex.com/change/rd119986a8678ba