Grain change history

Some Grain local captures were stuck processing in a major Recording Processing incident from 2026-06-11 23:12 UTC to 2026-06-23 16:14 UTC, lasting 11 days 17 h. Grain resolved the incident and reported that all hung local captures were recovered.

Grain reported a major Grain Recorder outage starting 2026-06-11 15:18 UTC, with scheduled bots delayed in joining and ad-hoc bot failures caused by an upstream provider outage. The incident lasted 6 h 6 min and was resolved at 2026-06-11 21:24 UTC, with no further bot-service degradation observed.

Grain Recorder had a major outage causing delayed bot joins and recording failures due to an upstream provider outage. It started at 2026-06-02 15:20 UTC and was resolved at 2026-06-03 16:49 UTC, lasting 25 h 29 min. Recordings stuck in processing required manual recovery; Grain confirmed all successfully captured recordings were recovered.

Grain Recorder had a critical outage affecting new bot recordings and local capture, starting 2026-03-24 15:32 UTC and resolved at 2026-03-24 19:58 UTC after 4 h 25 min. Grain attributed failures to an upstream provider outage and switched regions; instability ended, but cleanup of captured, unprocessed recordings was still underway.

Grain reported a major outage affecting Grain Recorder and Recording Processing from 2025-10-20 13:09 UTC to 2025-10-21 11:30 UTC, lasting 22 h 21 min. AWS instance-launch failures prevented scheduled and ad-hoc recorders from starting; Grain moved recording to a new AWS Region, then deployed further mitigations after patch side effects caused continued recorder failures.

Grain Recorder bots could not join meetings during a critical incident from 2025-07-28 13:21 UTC to 2025-07-28 16:47 UTC, lasting 3 h 26 min. An affected upstream provider deployed mitigations; Grain reported bots joining at normal rates by 07-28 14:11 UTC and later marked the incident resolved.

Grain Recorder experienced a major outage starting 2025-03-19 17:22 UTC, with failures scheduling and booting new recorders; some meetings may not have been captured. Mitigation initially stabilized recording, but errors rose again. The incident was resolved at 2025-05-29 16:24 UTC, with a recorded duration of 70 days 23 h.

Grain Recorder had a major degradation starting 2024-06-07 15:24 UTC, lasting 2 h 53 min until resolution at 2024-06-07 18:17 UTC. Recorder service scaling issues caused some bots to fail to join calls in a timely manner; Grain deployed a fix and reported no further issues.

Grain Recorder’s Meet recordings failed to capture video and speaker change events in a major incident starting 2024-05-21 15:44 UTC. A fix was deployed by 05-22 00:30 UTC; Grain reported no further issues and marked it resolved at 2024-06-07 18:18 UTC, with a recorded duration of 17 days 2 h.

Grain Web App experienced major application instability with a high rate of errors after a recent code deploy, from 2024-01-19 17:18 UTC to 2024-01-19 18:43 UTC (1 h 25 min). The team issued a hot patch and marked the incident resolved after observing no further instability.

Grain reported a critical outage starting 2023-12-19 15:33 UTC: Zoom blocked Grain Recorder from joining approximately 75% of calls. A temporary mitigation appeared successful; Grain later worked with Zoom to deploy a proper fix, reporting no further issues. The incident was resolved 2024-01-19 17:40 UTC, with a recorded duration of 31 days 2 h.

Grain Recorder experienced audio “popping” on some calls in a critical-impact incident starting 2023-11-28 18:34 UTC and lasting 20 days 23 h. A fix was deployed by 11-29 15:37 UTC, but affected recordings cannot be repaired. The incident was resolved at 2023-12-19 18:20 UTC, with no further instances reported since the fix.

Grain Recorder failed to join some Meet meetings in a critical incident starting 2023-08-08 06:51 UTC and resolved 2024-01-19 17:41 UTC, lasting 164 days 10 h. Grain suspected a new Google anti-spam error state and deployed an update to Google Meet authenticated recorders to work around Google's limitations.

Grain reported a critical Recording Processing incident: an Amazon outage delayed video processing from 2023-06-13 20:11 UTC to 2023-06-14 02:39 UTC, lasting 6 h 27 min. Grain reported that the Amazon outage appeared resolved and the video backlog was processing, then confirmed all systems were operating correctly.

Grain reported a critical outage starting 2023-05-10 17:12 UTC that lasted 1 h 1 min, with all functionality offline due to database issues, affecting Grain Recorder, Grain Web App and Recording Processing. Services were gradually re-enabled, and full functionality was restored at 2023-05-10 18:13 UTC.

Grain reported a critical Grain Recorder outage starting 2023-04-08 11:52 UTC: Zoom’s DNS outage prevented recording bots from joining meetings, causing most bot recordings to fail. Grain recommended Zoom Cloud Recordings; Zoom resolved the DNS outage, with no further recorder failures observed as of 15:09 UTC. The incident closed at 2023-04-08 18:36 UTC, lasting 6 h 44 min.