Changes / Grain / Failure to start recordings
Grain: Critical incident: Failure to start recordings
Summary
Grain Recorder had a critical outage affecting new bot recordings and local capture, starting 2026-03-24 15:32 UTC and resolved at 2026-03-24 19:58 UTC after 4 h 25 min. Grain attributed failures to an upstream provider outage and switched regions; instability ended, but cleanup of captured, unprocessed recordings was still underway.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Failure to start recordings
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2026-03-24 15:32 UTC
Before: not present
After: 2026-03-24 19:58 UTC
Before: not present
After: Grain Recorder
Before: not present
After: An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
Before: not present
After: The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
Before: not present
After: We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
Before: not present
After: We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
Before: not present
After: The team rolled back to the backup region due to continued higher than normal error rates.
Before: not present
After: Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Failure to start recordings (this change) Grain Recorder had a critical outage affecting new bot recordings and local capture, starting 2026-03-24 15:32 UTC and resolved at 2026-03-24 19:58 UTC after 4 h 25 min. Grain attributed failures to an upstream provider outage and switched regions; instability ended, but cleanup of captured, unprocessed recordings was still underway. | Major incident |