# Grain: Critical incident: Failure to start recordings

**Summary:** Grain Recorder had a critical outage affecting new bot recordings and local capture, starting 2026-03-24 15:32 UTC and resolved at 2026-03-24 19:58 UTC after 4 h 25 min. Grain attributed failures to an upstream provider outage and switched regions; instability ended, but cleanup of captured, unprocessed recordings was still underway. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-03-24 15:32 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/ykw7kxw2kq20](https://stspg.io/ykw7kxw2kq20)
- [Company: Grain](https://rewindex.com/company/grain.md)
- Versions: before (none: first archived) → after incident\_version:20/d7hw45pmsdww@f7e35d894af8
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Failure to start recordings
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-03-24 15:32 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-03-24 19:58 UTC
```

**6. Added: Affected components**

```diff
+ Grain Recorder
```

**7. Added: Update** (investigating) · 2026-03-24 15:32 UTC

```diff
+ An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
```

**8. Added: Update** (investigating) · 2026-03-24 15:40 UTC

```diff
+ The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
```

**9. Added: Update** (investigating) · 2026-03-24 17:02 UTC

```diff
+ We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
```

**10. Added: Update** (monitoring) · 2026-03-24 17:46 UTC

```diff
+ We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
```

**11. Added: Update** (monitoring) · 2026-03-24 18:31 UTC

```diff
+ The team rolled back to the backup region due to continued higher than normal error rates.
```

**12. Added: Update** (resolved) · 2026-03-24 19:58 UTC

```diff
+ Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-03-24 15:32 UTC | Critical incident: Failure to start recordings (this change) | outage |

All Grain changes: [timeline](https://rewindex.com/company/grain/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/iedf3307e526801 · Markdown: https://rewindex.com/change/iedf3307e526801.md · JSON: https://rewindex.com/api/v1/changes/iedf3307e526801 · Generated 2026-10-18T03:55:42.776Z