Rewindex

CompaniesLiveKitIncidents

LiveKit: Investigating reports of elevated API failures in Chicago region

MinorStatus: resolvededitedStarted Resolved (32 min)3 updatesstspg.io (opens in new tab)

Affected components: Regional Real Time Communication - US Central - Real Time CommunicationUS Central - Real Time CommunicationGlobal Real Time Communication

Timeline of updates

  1. investigatingEdited

    We are currently investigating reports of elevated API errors across our services in the Chicago region.

  2. monitoring

    A fix has been implemented as of 17:00 UTC and we are seeing the errors return back to baseline. We are continuing to monitor before marking this resolved.

  3. resolvedEdited

    Between 16:35-17:03 UTC, Chicago region experienced about 11% failure rates in API errors, failed agent dispatches, and issues connecting to rooms. We faced another wave of the same issue between 17:47-17:53 UTC where the failure rates were less than 10%. Room Service APIs called from other regions targeting to a room hosted in Chicago region were also affected during this period. API requests made with recent server SDK versions are automatically retried in other regions, so customers on those versions should not have seen these errors. The last related errors occurred at 17:53 UTC, and all affected services are operating normally since. We will share a detailed RCA with more information soon.

Changes Rewindex caught on this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

LiveKit profileRecent LiveKit incidentsMarkdownJSON