Grain: Outages with upstream providers causing instability
Grain reported a major outage affecting Grain Recorder and Recording Processing from 2025-10-20 13:09 UTC to 2025-10-21 11:30 UTC, lasting 22 h 21 min. AWS instance-launch failures prevented scheduled and ad-hoc recorders from starting; Grain moved recording to a new AWS Region, then deployed further mitigations after patch side effects caused continued recorder failures.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Grain RecorderRecording Processing
Timeline of updates
- monitoring
AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
- monitoring
We are continuing to monitor for any further issues.
- monitoring
AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
- monitoring
We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
- monitoring
The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
- resolved
This incident has been resolved.
Rewindex changes for this incident
- Major incident: Outages with upstream providers causing instability 2025-10-20 13:09 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Grain profile · Recent Grain incidents · Markdown · JSON