LiveKit: Investigating delayed session data ingestion on Cloud Dashboard
Affected components: Cloud Dashboard (cloud.livekit.io)
Timeline of updates
- investigating
We are currently investigating delayed session data ingestion on Cloud Dashboard. No services appear to be impacted and we don't expect any data to be lost.
- identified
Sessions data ingest remains delayed for about an hour. It's not impacting other dashboards. We've identified the source of the slow ingest and are working on a mitigation. We will update again in 30 mins.
- identified
We are continuing to work on a fix for this issue.
- identified
The amount of data we are ingesting in the sessions view is outpacing our ability to ingest, thus causing a delay in ingestion. We are pursuing a few mitigations right now including moving to a larger database instance to relieve the processing bottleneck.
- monitoring
Mitigations have been deployed and we are now catching up to realtime.
- resolved
This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).