crates.io: Request timeouts/Slow responses
AWS us-east-1 outage, Dec 22, 2021 · 40 vendors, 40 incidents
crates.io’s incident started in the same window without naming AWS (timing only, not evidence of a link). (timing only, posted −15 min vs the first vendor naming AWS)
Swipe sideways to see the whole diagram.
Mermaid source
flowchart LR
root["AWS us-east-1<br/>own incident not archived"]:::inferred
h1["Within an hour<br/>8 vendors, 6 blamed AWS<br/>Firebolt, Bitbucket, HubSpot, CircleCI +4"]:::blames
root --> h1
h4["1 to 4 hours later<br/>1 vendor<br/>Confluent"]:::names
root --> h4
me["crates.io (this incident)<br/>posted −15 min"]:::me
root -.->|"same window"| meFull event: timeline, evidence and all vendors →
Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.
Affected components: crates.io
Timeline of updates
- investigating
We are currently investigating an incident that causes slow responses and request timeouts on crates.io.
- investigating
We are continuing to investigate this issue.
- monitoring
An outage of one of our service providers impacts our primary database and one of our application dynos. Currently, write operations will fail (uploading new crates, ...) and downloads will not be counted. About 25% of all download requests hit the affected dyno and will fail with a timeout. Cargo itself should be mostly unaffected due to retries.
- monitoring
Our service provider is still working to restore full functionality, but unfortunately there is no ETA for full recovery at this time. We continue to monitor the situation.
- resolved
All of our affected infrastructure has recovered and full functionality has been restored. Some caches may require a bit of warming up, so some slow responses should still be expected.
Changes Rewindex caught on this incident
- Dec 22, 2021Major incident: Request timeouts/Slow responses
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.
How the archive is checked, and what it cannot prove.
First version captured 17:33 UTC
In the 2026-10-19 checkpoint, timestamped by FreeTSA and DigiCert on Oct 20, 00:41 UTC. Record hash
4b1aa4524b14f3c3…
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).