ChangesPyPI (Python Infrastructure)Python Infrastructure Outage
PyPI (Python Infrastructure): Critical incident: Python Infrastructure Outage
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what PyPI (Python Infrastructure) removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Python Infrastructure Outage
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2026-02-26 20:40 UTC
Before: not present
After: 2026-02-26 23:26 UTC
Before: not present
After: bugs.python.org, python.org - Backends, python.org - CDN, python.org - Downloads Backends, python.org - python.org - Backends, python.org - python.org - CDN, python.org - python.org - Downloads Backends, us.pycon.org
Before: not present
After: We are currently investigating this issue
Before: not present
After: The root cause has been identified as an erroneous scaling operation that affected some application workloads. All services have been restored and are coming back online. We are monitoring to confirm full recovery.
Before: not present
After: The incident has been resolved.
Before: not present
After: The incident has been resolved.
## Incident Report: Cluster-wide service outage
**Duration**: ~36 minutes \(20:04–20:40 UTC\)
**Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services.
What was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI.
**Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters.
The scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures.
**Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history.
Action items:
* Separate kubeconfig files for production vs local, rather than relying on context switching
* Research adding admission control or policies to prevent bulk scale-to-zero operations
* Document the infrastructure dependency chain and recovery runbook for future incidents
Jacob Coffee, PSF Infrastructure Team
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Thursday, February 26
1 incident change
- CriticalCritical incident
Python Infrastructure Outagethis change
All PyPI (Python Infrastructure) changesPyPI (Python Infrastructure) profileMarkdownJSON