# Grain: Bot Recording Failures

- Impact (vendor’s label): major
- Status: resolved
- Started: 2026-06-02 15:20 UTC · Resolved: 2026-06-03 16:49 UTC · Duration: 25 h
- Updates: 3 · Affected components: Grain Recorder
- [Vendor status page: https://stspg.io/qvj3wxclvfhl](https://stspg.io/qvj3wxclvfhl)
- [Company: Grain](https://rewindex.com/company/grain.md)

## Summary

Grain Recorder had a major outage causing delayed bot joins and recording failures due to an upstream provider outage. It started at 2026-06-02 15:20 UTC and was resolved at 2026-06-03 16:49 UTC, lasting 25 h 29 min. Recordings stuck in processing required manual recovery; Grain confirmed all successfully captured recordings were recovered. _(machine-written, sol)_
## Timeline of updates

- **investigating** 2026-06-02 15:20 UTC: Upstream provider is experiencing an outage causing out instability in our bot recorders including delayed joined times and recording failures. The team is investigating the scope and any potential mitigations.
- **monitoring** 2026-06-02 18:29 UTC: The outage has subsided. The team has identified a set of recordings currently stuck in processing state that require manual recovery and will be working to recover them as quickly as possible.
- **resolved** 2026-06-03 16:49 UTC: All recordings that were successfully captured have now been recovered. Future improvements around being more robust to upstream failures are being researched by the team.

## Rewindex changes for this incident

- 2026-06-02 15:20 UTC: [Major incident: Bot Recording Failures](https://rewindex.com/change/id83e54a8842403.md)


---
Canonical: https://rewindex.com/company/grain/incidents/bjg8vzc6vxcd · Markdown: https://rewindex.com/company/grain/incidents/bjg8vzc6vxcd.md · JSON: https://rewindex.com/api/v1/companies/grain/incidents/bjg8vzc6vxcd · Generated 2026-10-18T10:31:18.363Z