Companies / GitHub / incidents
GitHub: Incident with Issues, API Requests, Pull Requests, Actions, Pages, Git Operations, Webhooks, Packages and Codespaces
Timeline of updates
- resolved
On January 9 between 12:45 and 13:56 UTC, services in one of our three sites experienced elevated latency for connections. This led to a sustained period of timed out requests across a number of services, including but not limited to our git backend. An average of 5% and max of 10% of requests failed with a 5xx response or timed out during this period. This was caused by a combination of events that led to connection limits being hit in load balancer proxies in that site. An upgrade of hosts was in flight, which meant a subset of proxy hosts were draining and coming offline as the upgrade rolled through the fleet. A config change event also triggered a connection reset across all services in that site. These events are commonplace, but led to a spike in connection establishment events that led to the online proxy hosts hitting the connection limit. Upon further analysis, that limit was lower than it should have been. We have increased that limit to prevent this from recurring. We have also identified improvements to our monitoring of connection limits and behavior and changes to reduce the risk of proxy host upgrades leading to reduced capacity.
From GitHub’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.
Impact, status and times are the vendor’s own, as last archived. View on www.githubstatus.com (opens in new tab). GitHub profile · Recent GitHub incidents · Markdown · JSON