Changes / Grain / Bot Recording Failures
Grain: Major incident: Bot Recording Failures
Summary
Grain Recorder had a major outage causing delayed bot joins and recording failures due to an upstream provider outage. It started at 2026-06-02 15:20 UTC and was resolved at 2026-06-03 16:49 UTC, lasting 25 h 29 min. Recordings stuck in processing required manual recovery; Grain confirmed all successfully captured recordings were recovered.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Bot Recording Failures
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-06-02 15:20 UTC
Before: not present
After: 2026-06-03 16:49 UTC
Before: not present
After: Grain Recorder
Before: not present
After: Upstream provider is experiencing an outage causing out instability in our bot recorders including delayed joined times and recording failures. The team is investigating the scope and any potential mitigations.
Before: not present
After: The outage has subsided. The team has identified a set of recordings currently stuck in processing state that require manual recovery and will be working to recover them as quickly as possible.
Before: not present
After: All recordings that were successfully captured have now been recovered. Future improvements around being more robust to upstream failures are being researched by the team.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Bot Recording Failures (this change) Grain Recorder had a major outage causing delayed bot joins and recording failures due to an upstream provider outage. It started at 2026-06-02 15:20 UTC and was resolved at 2026-06-03 16:49 UTC, lasting 25 h 29 min. Recordings stuck in processing required manual recovery; Grain confirmed all successfully captured recordings were recovered. | Major incident |