rewindex

CompaniesLiveKitIncidents

LiveKit: Investigating failing calls and RoomService API failures in US Central/East and dashboard issues

MinorStatus: monitoringeditedStarted Not resolved8 updatesstatus.livekit.io (opens in new tab)

Affected components: Cloud Dashboard - cloud.livekit.ioRegional Real Time Communication - US Central - Real Time CommunicationRegional Real Time Communication - US East - Real Time CommunicationRegional SIP - US Central - SIPRegional SIP - US East - SIPUS Central - Real Time CommunicationUS Central - SIPUS East - Real Time CommunicationUS East - SIP

Timeline of updates

  1. investigating

    We are currently investigating an increased rate of failing outbound calls in US Central and US East. We will post another update with more information as soon as possible.

  2. investigating

    The LiveKit cloud dashboard is currently inaccessible. We are investigating this and will provide another update within 15 minutes.

  3. investigating

    We believe there is also impact to inbound calls in the affected regions. We are working on identifying the cause and will update again within 15 minutes.

  4. investigating

    We are currently attempting a mitigation. We will follow up with more information in the next 15 minutes.

  5. investigating

    We are still determining the full scope of impact, but we believe that the Room Service has also been impacted. We will update again within 15 minutes.

  6. monitoring

    We believe the mitigation was successful and are seeing services return to normal. The dashboard is accessible and error rates have gone down across services. We are continuing to monitor to make sure that the situation is resolved. We will update again as soon as possible.

  7. identified

    While the US regions still appear to be stable, we are tracking degraded services including SIP, RTC, and APIs in our Mexico region. We will update again as soon as possible.

  8. monitoring

    We've applied a mitigation for our Mexico region and services have returned to baseline. We are continuing to monitor and will update again within 15 minutes.

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Rewindex captured a newer version after the last checkpoint; the next run () commits it.

Impact, status and times are the vendor’s own, as last archived. View on status.livekit.io (opens in new tab).

LiveKit profileRecent LiveKit incidentsMarkdownJSON