Companies/GitHub/Incidents

GitHub: Incident with Issues, Webhooks and Actions

Timeline of updates

1. resolved2024-02-29 12:27 UTC

   On February 29, 2024, between 9:32 and 11:54 UTC, queuing in our background job service caused processing delays to Webhooks, Actions, and Issues. Nearly 95% of delays occurred between 11:05 and 11:27 UTC, with 5% during the remainder of the incident. During this incident, the following customer impacts occurred: 50% of webhooks experienced delays of up to 5m, 1% of webhooks experienced delays of 17m at peak; Actions: on average, 7% of customers experienced delays, with a peak of 44%; and many Issues saw a delay in appearing in searches. At 9:32 UTC our automated failover successfully routed traffic to a secondary cluster. But an improper restoration to primary at 10:32 UTC caused a significant increase in queued jobs until 11:21 UTC, when a correction was made and healthy services began burning down the backlog until full resolution. We have made improvements to the automation and reliability of our fallback process to prevent recurrence. We also have larger work already in progress to improve the overall reliability of our job processing platform.

From GitHub’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on www.githubstatus.com (opens in new tab).

GitHub profileRecent GitHub incidentsAll outages on February 29, 2024GitHub in February 2024MarkdownJSON