LiveKit: Hosted agent builds failing in US East
Affected components: Regional Cloud Agents - US East - Cloud AgentsUS East - Cloud Agents
Timeline of updates
- identified
New hosted agent builds are failing in US East. Existing agent workloads are not affected. We are currently deploying a mitigation and will follow up with more information shortly.
- monitoring
We have deployed a mitigation and builds are now successful. We will continue monitoring while we collect any further details to share.
- monitoring
The total build error rate has decreased significantly due to the mitigation, but a small percentage of builds in US East are still failing. We are continuing to monitor and will follow up with another update as soon as possible.
- resolved
We pushed a second mitigation at 22:14 UTC and all US East builds have been successful since 22:25 UTC. We will follow up with a postmortem describing the timeline, root cause, and follow ups as soon as possible.
- postmortem
### Summary An issue with the image registry which powers our hosted agents offering temporarily caused new hosted agent builds in US East to fail. Existing agent workloads were not affected, and no sessions or calls were impacted. ### Timeline \(UTC\) * 19:54 - Hosted agent builds begin failing in US East * 20:48 - Our team noticed the increased failure rate and began investigating * 21:04 - First mitigation deployed * 21:08 - Most builds succeeding * 22:14 - Second mitigation deployed, covering the remaining affected infrastructure * 22:24 - All builds succeeding ### Scope of Impact Only new hosted agent builds and deploys in US East were affected. Agents already running were unaffected, as were all sessions, calls, and other regions. Customers who were impacted would have seen their deploy fail with “unable to deploy agent: an error occurred while building your agent, please try again”. ### Mitigations and Follow-ups * We are improving alerting on build failures. * We are improving monitoring of our downstream image registries. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).