Daily: Networking issues
Affected components: APICore Call Experience
Timeline of updates
- investigating
We're investigating issues that may be causing problems with network connections between regions.
- investigating
We're continuing to investigate the source of meeting disruptions. Customers may be experiencing 'meeting moves' where a call session moves from one server to another, causing a 2-3 second disruption to the call. You may also see delays in receiving meeting.started and meeting.ended webhooks.
- identified
We've temporarily disabled the component that sends webhooks.
- identified
We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves. We'll post more information as soon as soon as it's available.
- monitoring
We've made some changes to the affected database, and our metrics and error rates have returned to normal. We've also re-enabled delivery of all webhooks, and we're monitoring for any further issues.
- monitoring
Our metrics have stayed at normal levels since our remediating actions about 30 minutes ago. We're continuing to monitor the platform while we discuss longer-term solutions to make absolutely sure we've addressed the root cause here.
- resolved
We've deployed an update that increases the throughput of the database that was the bottleneck in today's incident. We'll have more info about additional remediations and a postmortem for today's incident within the next few days.
Changes Rewindex caught on this incident
- Nov 25, 2024Major incident: Networking issues
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).