# Grain: Outages with upstream providers causing instability

- Impact (vendor’s label): major
- Status: resolved
- Started: 2025-10-20 13:09 UTC · Resolved: 2025-10-21 11:30 UTC · Duration: 22 h
- Updates: 6 · Affected components: Grain Recorder, Recording Processing
- [Vendor status page: https://stspg.io/mj55hsbxgv0m](https://stspg.io/mj55hsbxgv0m)
- [Company: Grain](https://rewindex.com/company/grain.md)

## Summary

Grain reported a major outage affecting Grain Recorder and Recording Processing from 2025-10-20 13:09 UTC to 2025-10-21 11:30 UTC, lasting 22 h 21 min. AWS instance-launch failures prevented scheduled and ad-hoc recorders from starting; Grain moved recording to a new AWS Region, then deployed further mitigations after patch side effects caused continued recorder failures. _(machine-written, sol)_
## Timeline of updates

- **monitoring** 2025-10-20 13:09 UTC: AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
- **monitoring** 2025-10-20 13:59 UTC: We are continuing to monitor for any further issues.
- **monitoring** 2025-10-20 14:55 UTC: AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
- **monitoring** 2025-10-20 16:07 UTC: We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
- **monitoring** 2025-10-20 18:03 UTC: The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
- **resolved** 2025-10-21 11:30 UTC: This incident has been resolved.

## Rewindex changes for this incident

- 2025-10-20 13:09 UTC: [Major incident: Outages with upstream providers causing instability](https://rewindex.com/change/i345276ad69c1da.md)


---
Canonical: https://rewindex.com/company/grain/incidents/2jsfx6gdfm2c · Markdown: https://rewindex.com/company/grain/incidents/2jsfx6gdfm2c.md · JSON: https://rewindex.com/api/v1/companies/grain/incidents/2jsfx6gdfm2c · Generated 2026-10-18T11:26:52.414Z