Changes / Codecov / Upload processing is severely delayed

Codecov: Major incident: Upload processing is severely delayed

Summary

Codecov Backend upload processing was severely delayed in a major-impact incident starting 2026-07-22 15:02 UTC and resolved at 2026-07-22 21:00 UTC, lasting 5 h 57 min. Codecov implemented a fix and reported the task backlog clearing, with affected users beginning to receive notifications.

Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.

Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: Upload processing is severely delayed
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2026-07-22 15:02 UTC
AddedResolved time#5
Before: not present
After: 2026-07-22 21:00 UTC
AddedAffected components#6
Before: not present
After: Codecov Backend
AddedUpdateinvestigating#7
Before: not present
After: We are currently investigating this issue.
AddedUpdateidentified#8
Before: not present
After: The issue has been identified and a fix is being implemented.
AddedUpdatemonitoring#9
Before: not present
After: A fix has been implemented and we are monitoring the results.
AddedUpdateinvestigating#10
Before: not present
After: We are currently investigating this issue.
AddedUpdateinvestigating#11
Before: not present
After: Changes previously implemented showed some improvement, but the issue remains and needs further investigation. Harness coverage team is currently working on additional investigation and changes.
AddedUpdateidentified#12
Before: not present
After: The issue has been identified and a fix is being implemented.
AddedUpdateidentified#13
Before: not present
After: We are continuing to work on a fix for this issue.
AddedUpdateidentified#14
Before: not present
After: The issue has been identified as a an issue with Codecov's redis instance. We are working on a fix.
AddedUpdatemonitoring#15
Before: not present
After: A fix has been implemented and we are monitoring the results.
AddedUpdatemonitoring#16
Before: not present
After: The backlog of tasks is clearing at a consistent rate. We will continue monitoring the situation.
AddedUpdatemonitoring#17
Before: not present
After: The backlog of tasks is continuing to clear, users affected in the early stages of the incident should expect to see their notifications.
AddedUpdateresolved#18
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Major incident: Upload processing is severely delayed (this change)
Codecov Backend upload processing was severely delayed in a major-impact incident starting 2026-07-22 15:02 UTC and resolved at 2026-07-22 21:00 UTC, lasting 5 h 57 min. Codecov implemented a fix and reported the task backlog clearing, with affected users beginning to receive notifications.
Major incident

All Codecov changes → · Codecov profile · Markdown · JSON