LiveKit: Relay issue in US East 2
Timeline of updates
- resolved
On Monday, around 4:37PT, the relay connections between US East 1 and US East 2 were lost, causing users connected to US East 2 to be unable to subscribe to tracks published to US East 1. This incident was caused by a bug with our track routing system (a component named Director). Director instances were restarted around 4:37PT. Typically when restarts take place, they would pick up where they've left off normally. However, on Monday, an edge case was triggered that caused Director instances to believe it was unable to connect to the other regions. This condition lasted around 4 minutes, before automatically recovering. Mitigation: We are making a couple of improvements in order to prevent similar situations from occurring: 1. We'll add additional safeguards in the code to prevent Director from triggering the disconnected state incorrectly. 2. From a process stand-point, this could have been prevented with staggered restarts, ensuring that each region only applies restarts/deploys during low traffic hours.
From LiveKit’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.
Impact, status and times are the vendor’s own, as last archived. View on status.livekit.io (opens in new tab).
Cite this
Status incident (vendor impact label: major) posted by LiveKit, started 2024-02-06 00:30 UTC, resolved after 0 min, 1 update(s).
- Title
- Relay issue in US East 2
- Vendor
- LiveKit
- Date
- URL
- https://rewindex.com/company/livekit/incidents/nmlm7nsb6mv9
LiveKit profileRecent LiveKit incidentsAll outages on February 6, 2024LiveKit in February 2024MarkdownJSON