ChangesPyPI (Python Infrastructure)PyPI Partial Outage
PyPI (Python Infrastructure): Major incident: PyPI Partial Outage
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what PyPI (Python Infrastructure) removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: PyPI Partial Outage
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-05-24 23:37 UTC
Before: not present
After: 2025-05-27 12:53 UTC
Before: not present
After: pypi.org - Backends, PyPI - pypi.org - Backends
Before: not present
After: We are working to investigate a partial outage in the PyPI backends.
Before: not present
After: We have identified a large increase in CDN cache misses causing excess load on our backends. We are working to determine the reason for this increase.
Before: not present
After: We have identified a flood of requests from a cluster using the `python-requests/2.32.3` User-Agent, and attempting to hit all JSON project/releasese en masse.
This causes excessive cache misses, which are overwhelming our backends. A temporary block has been put in place.
Before: not present
After: Our block is in place and systems are stable, but we are not satisfied with having to block such a generic user-agent. If you are aware of a system that began a mass crawl of PyPI's JSON release endpoints starting at around 22:05 UTC 2025-05-25, please contact admin@pypi.org.
Before: not present
After: We have removed the temporary block and are continuing to monitor.
Before: not present
After: We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Saturday, May 24, 2025
1 incident change
- MajorMajor incident
PyPI Partial Outagethis change
All PyPI (Python Infrastructure) changesPyPI (Python Infrastructure) profileMarkdownJSON