LiveKit: Elevated error rate on LiveKit Inference (google/gemma-4-31b-it)
Affected components: Global Inference
Timeline of updates
- identified
Between approximately 18:45 and 19:15 UTC today, a subset of LiveKit Inference requests using the google/gemma-4-31b-it model returned errors. Affected agent sessions would have seen an LLM request error on those requests. Other models and other LiveKit services were not affected. No action is needed on your part. If you continue to see errors, please reach out to support. We apologize for the disruption.
- identified
We are actively investigating elevated error rate on LiveKit Inference (google/gemma-4-31b-it)
- monitoring
A burst of requests increased the failure rate. Both primary and secondary model providers failed to fulfill the requests at the time; we are root-causing the issue. We are continuing to monitor.
- resolved
Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues.
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.
How the archive is checked, and what it cannot prove.
First version captured 17:03 UTC
In the 2026-10-19 checkpoint, timestamped by FreeTSA and DigiCert on Oct 20, 00:41 UTC. Record hash
53f0a90834641e86…
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).