Changes / Grain / High Rate of Recorder Failure
Grain: Major incident: High Rate of Recorder Failure
Summary
Grain Recorder experienced a major outage starting 2025-03-19 17:22 UTC, with failures scheduling and booting new recorders; some meetings may not have been captured. Mitigation initially stabilized recording, but errors rose again. The incident was resolved at 2025-05-29 16:24 UTC, with a recorded duration of 70 days 23 h.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: High Rate of Recorder Failure
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-03-19 17:22 UTC
Before: not present
After: 2025-05-29 16:24 UTC
Before: not present
After: Grain Recorder
Before: not present
After: Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
Before: not present
After: Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
Before: not present
After: We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: High Rate of Recorder Failure (this change) Grain Recorder experienced a major outage starting 2025-03-19 17:22 UTC, with failures scheduling and booting new recorders; some meetings may not have been captured. Mitigation initially stabilized recording, but errors rose again. The incident was resolved at 2025-05-29 16:24 UTC, with a recorded duration of 70 days 23 h. | Major incident |