Rewindex

CompaniesLiveKitIncidents

LiveKit: A percentage of cross-region calls between US West and US Central not completing

MinorStatus: resolvedStarted Resolved (1.1 h)3 updatesstspg.io (opens in new tab)

Timeline of updates

  1. investigating

    We are currently investigating automated alerting which triggered for degraded Room service performance in the US West and US Central regions beginning around 19:45 UTC. We are working to determine if there is any customer-facing impact. If there is impact, we don't currently have reason to believe that it is widespread. We will update again within 30 minutes.

  2. monitoring

    We have applied a mitigation to the affected regions and are now monitoring to ensure that performance signals return to baseline. We are still working on determining impact (including how users can determine if they were affected) and will share those details as soon as possible.

  3. resolved

    This incident has been resolved. Between 19:45 and 20:00 UTC, calls crossing the US West <-> US Central link experienced intermittent degradation in Room Service and SIP performance. The underlying network between the two regions had episodes of high packet loss during this time period. We rerouted traffic and performance has returned to baseline as of 22:00 UTC. We will follow up with a postmortem, which will include further details regarding scale and scope.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

LiveKit profileRecent LiveKit incidentsMarkdownJSON