# FireHydrant: Critical incident: Issue with Runbooks Attaching to Incidents

> Critical incident: Issue with Runbooks Attaching to Incidents. Detected 2026-11-04 by Rewindex: what changed on FireHydrant’s status page, with the before/after diff.

Canonical: https://rewindex.com/change/i95078a3fda487f
Last updated: 2026-11-04T09:08:40Z

**Summary:** _summary pending_
- Kind: Major incident · Label: degradation (model (Jev))
- Changed: 2024-07-23 16:30 UTC · Detected: 2026-11-04 09:08 UTC
- [Source: https://status.firehydrant.com/incidents/d80cd6ef-4299-4ab6-a5a8-986045372d87](https://status.firehydrant.com/incidents/d80cd6ef-4299-4ab6-a5a8-986045372d87)
- [Company: FireHydrant](https://rewindex.com/company/firehydrant.md)
- Versions: before (none: first archived) → after incident\_version:277/d80cd6ef-4299-4ab6-a5a8-986045372d87@34175875eb1b
- Change 1 of 1 for this incident

Incident page (full timeline): [Issue with Runbooks Attaching to Incidents](https://rewindex.com/company/firehydrant/incidents/d80cd6ef-4299-4ab6-a5a8-986045372d87.md)
## What changed

**1. Added: Title**

```diff
+ Issue with Runbooks Attaching to Incidents
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2024-07-23 16:30 UTC
```

**5. Added: Resolved time**

```diff
+ 2024-07-23 19:17 UTC
```

**6. Added: Update** (resolved) · 2024-07-23 16:40 UTC

```diff
+ We're investigating an issue with Runbooks failing to attach to incidents
```

**7. Added: Update** (resolved) · 2024-07-23 18:22 UTC

```diff
+ All backlogs have successfully cleared and Runbooks are functioning as expected
```

**8. Added: Update** (resolved) · 2024-07-23 19:17 UTC

```diff
+ FireHydrant experienced a prolonged delay of runbook executions due to a sudden outlier amount of traffic to our services. This traffic caused a shared worker queue to flood and go into a "spin lock" preventing other work from executing, such as runbook attachment. Existing incidents were not impacted from this incident, and our systems have been scaled to handle traffic spikes like the one experienced in the future.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2024-07-23 16:30 UTC | Critical incident: Issue with Runbooks Attaching to Incidents (this change) | degradation |

All FireHydrant changes: [timeline](https://rewindex.com/company/firehydrant/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._

## Cite this

Critical incident: Issue with Runbooks Attaching to Incidents on FireHydrant’s status page, changed 2024-07-23 16:30 UTC (change 1 of 1).

- Canonical URL: https://rewindex.com/change/i95078a3fda487f
- Citation: FireHydrant, “Critical incident: Issue with Runbooks Attaching to Incidents” (2024-07-23 16:30 UTC). Archived by Rewindex: https://rewindex.com/change/i95078a3fda487f (CC BY 4.0).
- Markdown link: `[FireHydrant: Critical incident: Issue with Runbooks Attaching to Incidents (2024-07-23)](https://rewindex.com/change/i95078a3fda487f)`


---
Canonical: https://rewindex.com/change/i95078a3fda487f · Markdown: https://rewindex.com/change/i95078a3fda487f.md · JSON: https://rewindex.com/api/v1/changes/i95078a3fda487f · Data licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) · Generated 2026-11-04T17:40:43.360Z