# Grain: Major incident: Bot Recording Failures

**Summary:** Grain Recorder had a major outage causing delayed bot joins and recording failures due to an upstream provider outage. It started at 2026-06-02 15:20 UTC and was resolved at 2026-06-03 16:49 UTC, lasting 25 h 29 min. Recordings stuck in processing required manual recovery; Grain confirmed all successfully captured recordings were recovered. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-06-02 15:20 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/qvj3wxclvfhl](https://stspg.io/qvj3wxclvfhl)
- [Company: Grain](https://rewindex.com/company/grain.md)
- Versions: before (none: first archived) → after incident\_version:20/bjg8vzc6vxcd@49472ce26f66
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Bot Recording Failures
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-06-02 15:20 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-06-03 16:49 UTC
```

**6. Added: Affected components**

```diff
+ Grain Recorder
```

**7. Added: Update** (investigating) · 2026-06-02 15:20 UTC

```diff
+ Upstream provider is experiencing an outage causing out instability in our bot recorders including delayed joined times and recording failures. The team is investigating the scope and any potential mitigations.
```

**8. Added: Update** (monitoring) · 2026-06-02 18:29 UTC

```diff
+ The outage has subsided. The team has identified a set of recordings currently stuck in processing state that require manual recovery and will be working to recover them as quickly as possible.
```

**9. Added: Update** (resolved) · 2026-06-03 16:49 UTC

```diff
+ All recordings that were successfully captured have now been recovered. Future improvements around being more robust to upstream failures are being researched by the team.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-06-02 15:20 UTC | Major incident: Bot Recording Failures (this change) | outage |

All Grain changes: [timeline](https://rewindex.com/company/grain/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/id83e54a8842403 · Markdown: https://rewindex.com/change/id83e54a8842403.md · JSON: https://rewindex.com/api/v1/changes/id83e54a8842403 · Generated 2026-10-18T02:55:05.039Z