Rewindex

CompaniesLiveKitIncidents

LiveKit: Intermittent errors for Google Gemini models via Inference

MinorStatus: resolvedStarted Resolved (5.9 h)3 updatesstspg.io (opens in new tab)

Affected components: Global Inference

Timeline of updates

  1. investigating

    We are currently investigating intermittent 429 errors for Google Gemini models routed through LiveKit Inference.

  2. identified

    We have identified the cause as an upstream limit on our Google Gemini API account. Automatic failover to an alternate Gemini deployment is in place and has reduced the impact, but a subset of requests to Google Gemini models may still intermittently return errors when failover capacity is exceeded. We are actively engaged with Google support to restore full capacity and will share further updates as we have them. Other models and providers remain unaffected.

  3. resolved

    This incident has been resolved. The errors were caused by an upstream issue at our model provider (Google) that incorrectly triggered an account-level usage cap on our Gemini API access, returning rate-limit errors for a subset of Gemini requests routed through LiveKit Inference. Google identified the cause, rolled back the change on their side, and raised our account limits to prevent recurrence. Gemini requests are now serving normally and error rates have returned to baseline. Other models and providers were unaffected throughout.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

LiveKit profileRecent LiveKit incidentsMarkdownJSON