Grain: Failure to start recordings
Grain Recorder had a critical outage affecting new bot recordings and local capture, starting 2026-03-24 15:32 UTC and resolved at 2026-03-24 19:58 UTC after 4 h 25 min. Grain attributed failures to an upstream provider outage and switched regions; instability ended, but cleanup of captured, unprocessed recordings was still underway.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Grain Recorder
Timeline of updates
- investigating
An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
- investigating
The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
- investigating
We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
- monitoring
We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
- monitoring
The team rolled back to the backup region due to continued higher than normal error rates.
- resolved
Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.
Rewindex changes for this incident
- Critical incident: Failure to start recordings 2026-03-24 15:32 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Grain profile · Recent Grain incidents · Markdown · JSON