Companies/PyPI (Python Infrastructure)/Incidents

PyPI (Python Infrastructure): Notice: PyPI Mass CDN Purge

Affected components: pypi.org - CDNPyPI - pypi.org - CDNpypi.org - BackendsPyPI - pypi.org - Backends

Timeline of updates

1. monitoring2022-07-29 15:19 UTC

   In order to resolve an issue in our CDN cache caused by a misconfiguration of Surrogate-Keys, PyPI will need to issue a "Purge All" of our CDN to clear out lingering cached objects that are not accessible to purge individually. This purge is very likely to impact performance when accessing PyPI for some time. In order to avoid purges like this in the future, our backends have been configured with Surrogate-Keys that will at least allow us to purge specific endpoint types rather than the entire service.

2. monitoring2022-07-29 15:30 UTC

   Purge All has been issued and we are monitoring as the backends attempt to meet the request volume. We anticipate that the service will continue to stabilize and reach steady state over the next 30-90 minutes.

3. monitoring2022-07-29 16:03 UTC

   Our backends have begun to stabilize as the caches are refilled. During the peak of the backend traffic from our CDN we recognized a few bottlenecks in our database layer and have provisioned additional storage IOPs in order to reduce impact of surges in the future.

4. monitoring2022-07-29 16:30 UTC

   Error rates have returned to baseline, and latencies continue to return to normal across our services. We will continue to monitor as the database upgrades complete and caches re-fill.

5. resolved2022-07-29 19:01 UTC

   Our CDN hit rate and latencies have recovered, this event is now resolved.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

PyPI (Python Infrastructure) profileRecent PyPI (Python Infrastructure) incidentsMarkdownJSON