Companies/GitHub/Incidents

GitHub: Incident with Pull Requests

Timeline of updates

1. resolved2023-10-05 16:42 UTC

   On October 5, 2023 at 13:40 UTC, our monitoring systems observed an increase in the time it was taking for Git pushes to become visible when viewing commits in Pull Requests. Under normal operating conditions, a series of asynchronous jobs runs in response to every push and within a few seconds applies a number of side-effects in Pull Requests such as requesting reviews, marking Pull Requests as merged, and showing new commits. During the incident, jobs were entering the queue faster than we could process them, resulting in processing delays as high as 75 seconds on average, and as much as 15 minutes in the worst case. About 10% of all Pull Request page loads were showing out-of-date data during this time. We had recently created a dedicated worker pool for processing these side-effects, with the goal of improving isolation between services and providing product teams with more direct control over critical parts of the system. We mitigated the incident by increasing capacity of the worker pool, which fully processed the backlog of delayed jobs, and returned everything to normal by 16:07 UTC. In response to this incident, we have adjusted our monitoring thresholds and improved our procedures for scaling up worker pools in response to increasing utilization.

From GitHub’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on www.githubstatus.com (opens in new tab).

GitHub profileRecent GitHub incidentsAll outages on October 5, 2023GitHub in October 2023MarkdownJSON