Changes / Grain / Zoom join failures
Grain: Critical incident: Zoom join failures
Summary
Grain reported a critical outage starting 2023-12-19 15:33 UTC: Zoom blocked Grain Recorder from joining approximately 75% of calls. A temporary mitigation appeared successful; Grain later worked with Zoom to deploy a proper fix, reporting no further issues. The incident was resolved 2024-01-19 17:40 UTC, with a recorded duration of 31 days 2 h.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Zoom join failures
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2023-12-19 15:33 UTC
Before: not present
After: 2024-01-19 17:40 UTC
Before: not present
After: Grain Recorder
Before: not present
After: Zoom has started blocking our Recorder from joining calls. We are in discussion with Zoom and we are attempting to work around the issue.
Before: not present
After: We are on a conference call with Zoom engineers trying to work around the issue
Before: not present
After: We've rolled out a potential short term mitigation and we're monitoring the solution.
Before: not present
After: Our temporary mitigation continues to appear to be successful. Zoom is still investigating the source of the issue as it appears to be a bug on their end causing our bot to be rejected by ~75% of calls. They have proposed a fix that we will be testing over the next 24 hours.
Before: not present
After: The team worked with Zoom over the holidays to deploy a proper fix. No further issues have been observed.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Zoom join failures (this change) Grain reported a critical outage starting 2023-12-19 15:33 UTC: Zoom blocked Grain Recorder from joining approximately 75% of calls. A temporary mitigation appeared successful; Grain later worked with Zoom to deploy a proper fix, reporting no further issues. The incident was resolved 2024-01-19 17:40 UTC, with a recorded duration of 31 days 2 h. | Major incident |