Daily: Major incident: Networking issues
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Daily removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Networking issues
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-11-25 16:48 UTC
Before: not present
After: 2024-11-25 19:26 UTC
Before: not present
After: API, Core Call Experience
Before: not present
After: We're investigating issues that may be causing problems with network connections between regions.
Before: not present
After: We're continuing to investigate the source of meeting disruptions. Customers may be experiencing 'meeting moves' where a call session moves from one server to another, causing a 2-3 second disruption to the call. You may also see delays in receiving meeting.started and meeting.ended webhooks.
Before: not present
After: We've temporarily disabled the component that sends webhooks.
Before: not present
After: We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves. We'll post more information as soon as soon as it's available.
Before: not present
After: We've made some changes to the affected database, and our metrics and error rates have returned to normal. We've also re-enabled delivery of all webhooks, and we're monitoring for any further issues.
Before: not present
After: Our metrics have stayed at normal levels since our remediating actions about 30 minutes ago. We're continuing to monitor the platform while we discuss longer-term solutions to make absolutely sure we've addressed the root cause here.
Before: not present
After: We've deployed an update that increases the throughput of the database that was the bottleneck in today's incident. We'll have more info about additional remediations and a postmortem for today's incident within the next few days.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Monday, November 25, 2024
1 incident change
- MajorMajor incident
Networking issuesthis change