Grain: Webhook consumers failing causing app failures
Affected components: Grain RecorderRecording Processing
Timeline of updates
- investigating
The team is actively rolling back a failed update for its webhook consumers. Webhooks are currently failing to process leading to UI to fail to update for bots and recording processing. No data loss has occurred.
- monitoring
The team has deployed a rollback and webhooks are correctly process. The team is determining if any recording process needs manual intervention.
- monitoring
The team is manually reprocessing all dangling completed recordings as they become available. All new recordings are processing correctly.
Changes Rewindex caught on this incident
- Major incident: Webhook consumers failing causing app failures
- Incident updated: Webhook consumers failing causing app failures
- Incident updated: Webhook consumers failing causing app failures
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
Rewindex captured this after the last checkpoint; the next run () commits it.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).