FireHydrant: Issue with Runbooks Attaching to Incidents
Affected components: Runbook Execution
Timeline of updates
- resolved
We're investigating an issue with Runbooks failing to attach to incidents
- resolved
All backlogs have successfully cleared and Runbooks are functioning as expected
- resolved
FireHydrant experienced a prolonged delay of runbook executions due to a sudden outlier amount of traffic to our services. This traffic caused a shared worker queue to flood and go into a "spin lock" preventing other work from executing, such as runbook attachment. Existing incidents were not impacted from this incident, and our systems have been scaled to handle traffic spikes like the one experienced in the future.
Changes Rewindex caught on this incident
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.
Impact, status and times are the vendor’s own, as last archived. View on status.firehydrant.com (opens in new tab).
Cite this
Status incident (vendor impact label: critical) posted by FireHydrant, started 2024-07-23 16:30 UTC, resolved after 2.8 h, 3 update(s).
- Title
- Issue with Runbooks Attaching to Incidents
- Vendor
- FireHydrant
- Date
- URL
- https://rewindex.com/company/firehydrant/incidents/d80cd6ef-4299-4ab6-a5a8-986045372d87
FireHydrant profileRecent FireHydrant incidentsAll outages on July 23, 2024FireHydrant in July 2024MarkdownJSON