Grain: High Rate of Recorder Failure
Grain Recorder experienced a major outage starting 2025-03-19 17:22 UTC, with failures scheduling and booting new recorders; some meetings may not have been captured. Mitigation initially stabilized recording, but errors rose again. The incident was resolved at 2025-05-29 16:24 UTC, with a recorded duration of 70 days 23 h.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Grain Recorder
Timeline of updates
- investigating
Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
- monitoring
Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
- monitoring
We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
- resolved
This incident has been resolved.
Rewindex changes for this incident
- Major incident: High Rate of Recorder Failure 2025-03-19 17:22 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Grain profile · Recent Grain incidents · Markdown · JSON