# Grain: High Rate of Recorder Failure

- Impact (vendor’s label): major
- Status: resolved
- Started: 2025-03-19 17:22 UTC · Resolved: 2025-05-29 16:24 UTC · Duration: 71 d
- Updates: 4 · Affected components: Grain Recorder
- [Vendor status page: https://stspg.io/y329z6pjzzj9](https://stspg.io/y329z6pjzzj9)
- [Company: Grain](https://rewindex.com/company/grain.md)

## Summary

Grain Recorder experienced a major outage starting 2025-03-19 17:22 UTC, with failures scheduling and booting new recorders; some meetings may not have been captured. Mitigation initially stabilized recording, but errors rose again. The incident was resolved at 2025-05-29 16:24 UTC, with a recorded duration of 70 days 23 h. _(machine-written, sol)_
## Timeline of updates

- **investigating** 2025-03-19 17:22 UTC: Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
- **monitoring** 2025-03-19 18:13 UTC: Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
- **monitoring** 2025-03-19 19:14 UTC: We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
- **resolved** 2025-05-29 16:24 UTC: This incident has been resolved.

## Rewindex changes for this incident

- 2025-03-19 17:22 UTC: [Major incident: High Rate of Recorder Failure](https://rewindex.com/change/ic815891f73d2e2.md)


---
Canonical: https://rewindex.com/company/grain/incidents/8z32m4y6v3gh · Markdown: https://rewindex.com/company/grain/incidents/8z32m4y6v3gh.md · JSON: https://rewindex.com/api/v1/companies/grain/incidents/8z32m4y6v3gh · Generated 2026-10-18T11:40:05.274Z