rewindex

ChangesFireHydrantIssue with Runbooks Attaching to Incidents

FireHydrant: Critical incident: Issue with Runbooks Attaching to Incidents

Major incidentdegradation (model (Jev), 51%)Changed Detected status.firehydrant.com (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what FireHydrant removed; blue is what they added. Muted text is unchanged context.

AddedTitle#1
Before: not present
After: Issue with Runbooks Attaching to Incidents
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2024-07-23 16:30 UTC
AddedResolved time#5
Before: not present
After: 2024-07-23 19:17 UTC
AddedUpdateresolved#6
Before: not present
After: We're investigating an issue with Runbooks failing to attach to incidents
AddedUpdateresolved#7
Before: not present
After: All backlogs have successfully cleared and Runbooks are functioning as expected
AddedUpdateresolved#8
Before: not present
After: FireHydrant experienced a prolonged delay of runbook executions due to a sudden outlier amount of traffic to our services. This traffic caused a shared worker queue to flood and go into a "spin lock" preventing other work from executing, such as runbook attachment. Existing incidents were not impacted from this incident, and our systems have been scaled to handle traffic spikes like the one experienced in the future.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.

How the archive is checked, and what it cannot prove.

Rewindex captured this after the last checkpoint; the next run () commits it.

This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.

History of this incident

Tuesday, July 23, 2024

1 incident change

Cite this

Critical incident: Issue with Runbooks Attaching to Incidents on FireHydrant’s status page, changed 2024-07-23 16:30 UTC (change 1 of 1).

Title
Critical incident: Issue with Runbooks Attaching to Incidents
Vendor
FireHydrant
Date
URL
https://rewindex.com/change/i95078a3fda487f

All FireHydrant changesFireHydrant profileMarkdownJSON