Changes / Grain / Outages with upstream providers causing instability
Grain: Major incident: Outages with upstream providers causing instability
Summary
Grain reported a major outage affecting Grain Recorder and Recording Processing from 2025-10-20 13:09 UTC to 2025-10-21 11:30 UTC, lasting 22 h 21 min. AWS instance-launch failures prevented scheduled and ad-hoc recorders from starting; Grain moved recording to a new AWS Region, then deployed further mitigations after patch side effects caused continued recorder failures.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Outages with upstream providers causing instability
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-10-20 13:09 UTC
Before: not present
After: 2025-10-21 11:30 UTC
Before: not present
After: Grain Recorder, Recording Processing
Before: not present
After: AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
Before: not present
After: We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
Before: not present
After: The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Outages with upstream providers causing instability (this change) Grain reported a major outage affecting Grain Recorder and Recording Processing from 2025-10-20 13:09 UTC to 2025-10-21 11:30 UTC, lasting 22 h 21 min. AWS instance-launch failures prevented scheduled and ad-hoc recorders from starting; Grain moved recording to a new AWS Region, then deployed further mitigations after patch side effects caused continued recorder failures. | Major incident |