Changes / Grain / Recorder scaling issues
Grain: Major incident: Recorder scaling issues
Summary
Grain Recorder had a major degradation starting 2024-06-07 15:24 UTC, lasting 2 h 53 min until resolution at 2024-06-07 18:17 UTC. Recorder service scaling issues caused some bots to fail to join calls in a timely manner; Grain deployed a fix and reported no further issues.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 75%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Recorder scaling issues
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-06-07 15:24 UTC
Before: not present
After: 2024-06-07 18:17 UTC
Before: not present
After: Grain Recorder
Before: not present
After: We are seeing issues with the scaling of the recorder service causing some bots to fail to join calls in a timely manner.
Before: not present
After: A fix was deployed and we have not seen further issues.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Recorder scaling issues (this change) Grain Recorder had a major degradation starting 2024-06-07 15:24 UTC, lasting 2 h 53 min until resolution at 2024-06-07 18:17 UTC. Recorder service scaling issues caused some bots to fail to join calls in a timely manner; Grain deployed a fix and reported no further issues. | Major incident |