ChangesLiveKitObservability upload and Agent Simulations failures
LiveKit: Incident updated: Observability upload and Agent Simulations failures
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what LiveKit removed; blue is what they added. Muted text is unchanged context.
Before: resolved
After: postmortem
Before: not present
After: **Root Cause**
During a security rotation of access credentials, we revoked a storage credential that we believed was no longer in use. It was still being used by our observability platform, which stores the data behind the dashboard and the Analytics API, the Agent Insights data that agents upload at the end of each session, and by the service that serves that data back for downloads and Agent Simulations. Once the credential was revoked, uploads failed in all regions.
Restoring service took longer than a credential swap normally would for three reasons:
* New credentials had to be set up, and each affected service had to be updated to use them.
* The new credentials took longer than expected to become available in all regions \(about two hours in our US East region\).
* Once the observability platform was working again, we found additional services that also needed updating. The incident was resolved once they were updated.
**Timeline \(UTC\)**
* 2026-10-13 08:13 - A storage credential is revoked during a security rotation. Observability uploads begin failing in all regions.
* 08:15 - Agent Insights transcript and audio downloads and Agent Simulations begin failing.
* 09:39 - Our team detects the failures and opens an incident.
* 10:45 - The revoked credential is identified as the cause.
* 11:16 - Affected services are reconfigured to use a new credential.
* 12:40 - The new credential takes effect in the last region. Observability uploads recover in all regions.
* 13:13 - Agent Insights downloads recover.
* 13:26 - Agent Simulations recover.
* 13:30 - Incident resolved.
* 2026-10-17 - Recovery of observability data from the impact window completes \(see below\).
**Scope of Impact**
* **Observability uploads \(08:13 to 12:40 UTC, all regions\):** Observability data written during this window, including the data behind the dashboard and the Analytics API, failed to store. Nearly all agent sessions that ended in this window also failed to upload their Agent Insights data, so in Agent Insights these sessions were missing from the sessions list, or shown as "In progress". Agent sessions themselves ran normally, and media and calls were not affected.
* **Data recovery:** In our resolved update we said all Agent Insights data from this window was unrecoverable. We have since restored most of it. **Session recordings and transcripts from this window are permanently lost.** All other observability data for these sessions has been restored and is visible in the dashboard and through the Analytics API.
* **Agent Insights downloads \(08:15 to 13:13 UTC\):** Downloading transcripts and audio for existing sessions failed during the incident window.
* **Agent Simulations \(08:15 to 13:26 UTC\):** Simulation runs couldn't load conversation transcripts, so they finished without a summary or verdict, or were cancelled. Runs from this window need to be re-run.
* **Billing:** Agent Insights usage is only recorded after a successful upload, so failed uploads were **not** billed.
**Detection and Response**
We detected this issue about 85 minutes after it began. We did not have alerting on upload failures to observability storage, so it was caught when an engineer noticed elevated errors rather than by an automated alert. We also initially marked the incident resolved once uploads recovered, before confirming that Agent Simulations and downloads had recovered too.
**Mitigations and Follow-ups**
* We have added alerting that pages our on-call team when observability storage uploads fail, along with a continuous check that stored data can still be read and downloaded and an end-to-end test that verifies our storage credentials still work.
* We have built tooling to replay and restore observability data from a past window, which we used to recover data from this incident.
* We are moving these services to dedicated service credentials managed with our infrastructure and auditing other services for the same pattern.
* We have updated our credential rotation process to verify every consumer of a credential before revoking it, and to account for regional propagation delays when rolling out replacements.
* We are adding redundancy so a single failed credential can no longer stop observability uploads.
We sincerely apologize for the disruption. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.
Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.
How the archive is checked, and what it cannot prove.
Before captured
In the 2026-10-19 checkpoint, timestamped by FreeTSA and DigiCert on . Record hash
d7fde4f16b9c3bb9…After captured
In the 2026-10-21 checkpoint, timestamped by FreeTSA and DigiCert on . Record hash
9fcc7ab38aba9724…
History of this incident
Wednesday, October 21
1 incident change
- Normal progressIncident
Incident updateObservability upload and Agent Simulations failuresthis change
Tuesday, October 13
1 incident change
- OutageCritical incident
Cite this
Incident updated: Observability upload and Agent Simulations failures on LiveKit’s status page, changed 2026-10-21 23:53 UTC (change 2 of 2).
- Title
- Incident updated: Observability upload and Agent Simulations failures
- Vendor
- LiveKit
- Date
- URL
- https://rewindex.com/change/r8fec9709fbd7bf