Rewindex

ChangesPyPI (Python Infrastructure)Object storage provider outage

PyPI (Python Infrastructure): Critical incident: Object storage provider outage

Major incidentcritical (vendor)Changed Detected stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what PyPI (Python Infrastructure) removed; blue is what they added. Muted text is unchanged context.

AddedTitle#1
Before: not present
After: Object storage provider outage
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2023-06-21 21:38 UTC
AddedResolved time#5
Before: not present
After: 2023-06-22 01:59 UTC
AddedAffected components#6
Before: not present
After: files.pythonhosted.org - Files, PyPI - files.pythonhosted.org - Files
AddedUpdateinvestigating#7
Before: not present
After: We are currently investigating an issue with our object storage provider.
AddedUpdateidentified#8
Before: not present
After: We have identified an issue that is impacting latencies for files.pythonhosted.org, resulting in timeouts for a portion of requests.
AddedUpdateidentified#9
Before: not present
After: We're in communication with the object storage provider and are waiting for them to rollout a fix.
AddedUpdateidentified#10
Before: not present
After: We are also implementing our own change to guard against excessive latencies from the provider.
AddedUpdatemonitoring#11
Before: not present
After: Our mitigation has decreased overall latencies, but they remain elevated over normal operation. As our provider's service recovers we expect to see a return to baseline.
AddedUpdateresolved#12
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.

How the archive is checked, and what it cannot prove.

Rewindex captured this after the last checkpoint; the next run (00:41 UTC) commits it.

This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.

History of this incident

Wednesday, June 21, 2023

1 incident change

All PyPI (Python Infrastructure) changesPyPI (Python Infrastructure) profileMarkdownJSON