Companies/LiveKit/Incidents

LiveKit: Disconnections and failed connections in multiple regions

Affected components: Europe Central - Real Time CommunicationEurope South - Real Time CommunicationIsrael - Real Time CommunicationRegional Ingress - US Central - IngressRegional Ingress - US East - IngressRegional Ingress - US West - IngressRegional Real Time Communication - Europe Central - Real Time CommunicationRegional Real Time Communication - Europe South - Real Time CommunicationRegional Real Time Communication - Israel - Real Time CommunicationRegional Real Time Communication - Saudi Arabia - Real Time CommunicationRegional Real Time Communication - UAE - Real Time CommunicationRegional Real Time Communication - US Central - Real Time CommunicationRegional Real Time Communication - US East - Real Time CommunicationRegional Real Time Communication - US West - Real Time CommunicationSaudi Arabia - Real Time CommunicationUAE - Real Time CommunicationUS Central - IngressUS Central - Real Time CommunicationUS East - IngressUS East - Real Time CommunicationUS West - IngressUS West - Real Time Communication

Timeline of updates

1. resolved2026-10-31 07:43 UTC

   This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly.
   During this window:
   - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join.
   - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed.
   - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed.
   The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC.
   Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.

2. postmortem2026-11-03 13:59 UTC

   Summary
   On October 31, 2026, between 07:10 and 07:43 UTC, an uncommon data pattern surfaced a latent software bug which caused media servers in several regions to restart repeatedly. A small percentage of participants were disconnected and the vast majority automatically reconnected. Impact was concentrated in North America. We stopped the trigger at 07:43 UTC and deployed a fix to all regions by 08:49 UTC.
   We apologize for the disruption this caused.
   Root Cause
   Applications can open their own data channel and send raw data over it, outside of LiveKit's standard data message format. Media servers forward this raw data to other participants in the room, wrapping it with the sender's identity so that receivers know who sent it. When a room spans multiple regions, the media servers hosting it also forward this data to each other. That server-to-server step sent the raw data without the sender's identity, and the receiving server assumed the identity would be present, which caused it to crash. This gap in our server-to-server protocol had been present for some time and was not introduced by a recent deployment.
   Starting at 07:10 UTC, one application sent raw data over its own data channel in a room that spanned multiple regions. Every server that received the forwarded message crashed at the same time, while the server the sender was connected to kept running and kept sending. Participants on the crashed servers automatically reconnected to other servers. Those servers then began hosting the room, received the next message, and crashed as well. This cycle repeated until we stopped the trigger at 07:43 UTC. In total, media servers in 12 of our data centers restarted, some of them up to 10 times.
   Timeline (UTC)
   2026-10-31 07:10 - An application begins sending raw data over its own data channel in a multi-region room. Media servers in US East and Canada restart within seconds.
   07:13 - Our monitoring detects the restarts and an incident is opened automatically.
   07:10 to 07:43 - Media servers in 12 data centers restart repeatedly as participants reconnect to new servers.
   07:43 - We block the traffic triggering the bug. There are no further restarts after this point, and customer impact ends.
   07:46 - A fix for the bug is ready.
   08:04 - The fix begins rolling out, starting with a small set of regions.
   08:49 - The fix is deployed to all regions.
   Scope of Impact
   Realtime sessions:
   Across all regions, about 1.5% were disconnected more than once within five minutes.
   In US East, US Central, US West, Canada and Mexico, about 6% were disconnected more than once within five minutes.
   The median time from disconnect to rejoin was about 3 seconds.
   Agents:
   Agents were disconnected and reconnected the same way as other participants, with the same median reconnect time of about 3 seconds. An agent that reconnects keeps its conversation state, so most conversations continued after a short gap. Fewer than 1% of active agent sessions did not recover.
   Ingress, Egress and SIP:
   Most active WHIP ingress streams were interrupted, and reconnects typically took about 40 seconds. Some WHIP ingress update and delete requests also failed.
   About 5% of active egress sessions were disconnected. Most reconnected within a few seconds.
   About 1% of SIP participants were disconnected, and half of them rejoined in under a second. Some SIP call transfers also failed.
   APIs:
   About 1.3% of Room Service API requests failed during the window. ListParticipants and GetParticipant were the most affected.
   How to tell if you were affected: participants in affected sessions saw a reconnecting state, or a brief loss of audio and video. In session analytics, an affected participant appears to leave and rejoin the same room with the same identity between 07:10 and 07:43 UTC.
   Detection and Response
   Our monitoring detected the server restarts about 3 minutes after they began, and an incident was opened automatically. Identifying the specific room and traffic triggering the restarts took about 30 minutes. Once we found it, we blocked the trigger at 07:43 UTC and the restarts stopped immediately. We deployed a fix to all regions by 08:49 UTC.
   We did not post to our status page while the incident was in progress, and published a notice only after it was resolved. Our original post mistakenly sent notifications as we initially over-estimated the scope of impact.
   Mitigations and Follow-ups
   We fixed the bug. Media servers now handle forwarded data that arrives without the sender's identity instead of crashing. This fix was deployed to all regions by 08:49 UTC.
   Raw data forwarded between media servers now includes the sender's identity, so receiving servers always know who sent it.
   We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

Changes Rewindex caught on this incident

• Nov 3Incident updated: Disconnections and failed connections in multiple regions

Impact, status and times are the vendor’s own, as last archived. View on status.livekit.io (opens in new tab).

LiveKit profileRecent LiveKit incidentsAll outages on October 31, 2026LiveKit in October 2026MarkdownJSON