Changescrates.ioIntermittent timeouts on API endpoints
crates.io: Major incident: Intermittent timeouts on API endpoints
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what crates.io removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Intermittent timeouts on API endpoints
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-02-14 19:30 UTC
Before: not present
After: 2024-02-15 02:43 UTC
Before: not present
After: crates.io
Before: not present
After: Some crates.io endpoints are timing out at present, including the summary route that drives the crates.io home page. We are investigating.
Before: not present
After: (If this looks suspiciously similar to https://status.crates.io/incidents/t49v2pfpv0vl, the same issue reappeared on the summary endpoint literally within seconds of resolving that incident. C'est la vie.)
Before: not present
After: We believe this issue is being caused by excess database load related to a bug fix deployed earlier today around download counting. This bug fix has caused the normal background processing of per-crate download count totals to take significantly longer and require more resources than usual.
We will shortly be temporarily disabling the summary endpoint to alleviate some of the load on the database.
Before: not present
After: The long running background job has completed, and response times for the summary endpoint have returned to normal.
The next invocation of the relevant background job will be at 00:30 UTC (so in just under 20 minutes); we will be monitoring that closely to see if any problems resurface at that point.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: crates.io has returned to normal service with the processing of the download count backlog from earlier and the completion of the long running background jobs. Investigations will continue during normal hours for the crates.io team to ascertain what is causing elevated database load.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Wednesday, February 14, 2024
1 incident change
- MajorMajor incident
Intermittent timeouts on API endpointsthis change