# Grain: Major incident: Outages with upstream providers causing instability

**Summary:** Grain reported a major outage affecting Grain Recorder and Recording Processing from 2025-10-20 13:09 UTC to 2025-10-21 11:30 UTC, lasting 22 h 21 min. AWS instance-launch failures prevented scheduled and ad-hoc recorders from starting; Grain moved recording to a new AWS Region, then deployed further mitigations after patch side effects caused continued recorder failures. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2025-10-20 13:09 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/mj55hsbxgv0m](https://stspg.io/mj55hsbxgv0m)
- [Company: Grain](https://rewindex.com/company/grain.md)
- Versions: before (none: first archived) → after incident\_version:20/2jsfx6gdfm2c@e3794fa037f6
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Outages with upstream providers causing instability
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2025-10-20 13:09 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-10-21 11:30 UTC
```

**6. Added: Affected components**

```diff
+ Grain Recorder, Recording Processing
```

**7. Added: Update** (monitoring) · 2025-10-20 13:09 UTC

```diff
+ AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
```

**8. Added: Update** (monitoring) · 2025-10-20 13:59 UTC

```diff
+ We are continuing to monitor for any further issues.
```

**9. Added: Update** (monitoring) · 2025-10-20 14:55 UTC

```diff
+ AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
```

**10. Added: Update** (monitoring) · 2025-10-20 16:07 UTC

```diff
+ We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
```

**11. Added: Update** (monitoring) · 2025-10-20 18:03 UTC

```diff
+ The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
```

**12. Added: Update** (resolved) · 2025-10-21 11:30 UTC

```diff
+ This incident has been resolved.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-10-20 13:09 UTC | Major incident: Outages with upstream providers causing instability (this change) | outage |

All Grain changes: [timeline](https://rewindex.com/company/grain/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i345276ad69c1da · Markdown: https://rewindex.com/change/i345276ad69c1da.md · JSON: https://rewindex.com/api/v1/changes/i345276ad69c1da · Generated 2026-10-18T02:55:10.918Z