rewindex

ChangesGitHubIncident with Webhooks and Actions

GitHub: Incident silently edited: Incident with Webhooks and Actions

Silent editNot classified yetChanged www.githubstatus.com (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Discussed on Hacker News

Threads are matched automatically by vendor name and timing; points and comments refresh every 30 minutes. Method and limits.

Incident page: the full timeline of updates

What changed

Struck in red is what GitHub removed; blue is what they added. Muted text is unchanged context.

ChangedUpdateresolved#1
Before: This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
After: On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers.

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdayThursday, November 12Thu, Nov 12

1 rewrite

Monday, November 9

6 incident changes

Cite this

Incident silently edited: Incident with Webhooks and Actions on GitHub’s status page, changed 2026-11-12 19:13 UTC (change 7 of 7).

Title
Incident silently edited: Incident with Webhooks and Actions
Vendor
GitHub
Date
URL
https://rewindex.com/change/r51689230c3a280

All GitHub changesGitHub profileMarkdownJSON