# Grain: Major incident: High Rate of Recorder Failure

**Summary:** Grain Recorder experienced a major outage starting 2025-03-19 17:22 UTC, with failures scheduling and booting new recorders; some meetings may not have been captured. Mitigation initially stabilized recording, but errors rose again. The incident was resolved at 2025-05-29 16:24 UTC, with a recorded duration of 70 days 23 h. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2025-03-19 17:22 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/y329z6pjzzj9](https://stspg.io/y329z6pjzzj9)
- [Company: Grain](https://rewindex.com/company/grain.md)
- Versions: before (none: first archived) → after incident\_version:20/8z32m4y6v3gh@8905a1798f02
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ High Rate of Recorder Failure
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2025-03-19 17:22 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-05-29 16:24 UTC
```

**6. Added: Affected components**

```diff
+ Grain Recorder
```

**7. Added: Update** (investigating) · 2025-03-19 17:22 UTC

```diff
+ Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
```

**8. Added: Update** (monitoring) · 2025-03-19 18:13 UTC

```diff
+ Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
```

**9. Added: Update** (monitoring) · 2025-03-19 19:14 UTC

```diff
+ We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
```

**10. Added: Update** (resolved) · 2025-05-29 16:24 UTC

```diff
+ This incident has been resolved.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-03-19 17:22 UTC | Major incident: High Rate of Recorder Failure (this change) | outage |

All Grain changes: [timeline](https://rewindex.com/company/grain/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ic815891f73d2e2 · Markdown: https://rewindex.com/change/ic815891f73d2e2.md · JSON: https://rewindex.com/api/v1/changes/ic815891f73d2e2 · Generated 2026-10-18T02:54:51.141Z