Changes / Grain / Bots failing to join calls

Grain: Critical incident: Bots failing to join calls

Summary

Grain Recorder bots could not join meetings during a critical incident from 2025-07-28 13:21 UTC to 2025-07-28 16:47 UTC, lasting 3 h 26 min. An affected upstream provider deployed mitigations; Grain reported bots joining at normal rates by 07-28 14:11 UTC and later marked the incident resolved.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Bots failing to join calls
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2025-07-28 13:21 UTC
AddedResolved time#5
Before: not present
After: 2025-07-28 16:47 UTC
AddedAffected components#6
Before: not present
After: Grain Recorder
AddedUpdateinvestigating#7
Before: not present
After: We are currently investigating an issue that is causing bots to be unable to join meetings.
AddedUpdateidentified#8
Before: not present
After: Our affected upstream provider has identified the issue and is in the process of deploying mitigations.
AddedUpdatemonitoring#9
Before: not present
After: Our affected upstream provider has deployed mitigations and we are now observing bots joining meetings at normal rates. We will continue to monitor for stability.
AddedUpdateresolved#10
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Critical incident: Bots failing to join calls (this change)
Grain Recorder bots could not join meetings during a critical incident from 2025-07-28 13:21 UTC to 2025-07-28 16:47 UTC, lasting 3 h 26 min. An affected upstream provider deployed mitigations; Grain reported bots joining at normal rates by 07-28 14:11 UTC and later marked the incident resolved.
Major incident

All Grain changes → · Grain profile · Markdown · JSON