Changes / Travis CI / Queued jobs are taking time to process

Travis CI: Critical incident: Queued jobs are taking time to process

Summary

Travis CI reported critical delays processing queued builds, affecting Linux, Windows and Mac build processing. The incident started at 2021-06-21 15:05 UTC and resolved at 2021-06-22 08:05 UTC, lasting 16 h 59 min. Travis CI identified the problem and worked on a fix but provided no cause or fix details.

Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.

Major incidentLabel: degradation (model (Jev), 99%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Queued jobs are taking time to process
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2021-06-21 15:05 UTC
AddedResolved time#5
Before: not present
After: 2021-06-22 08:05 UTC
AddedAffected components#6
Before: not present
After: Builds Processing - Linux and Windows Builds, Builds Processing - Mac Builds, Linux and Windows Builds
AddedUpdateinvestigating#7
Before: not present
After: Some builds may have trouble on the queued stage. We are currently investigating the issue.
AddedUpdateidentified#8
Before: not present
After: We have identified the problem and a fix is in process.
AddedUpdateidentified#9
Before: not present
After: We are continuing to work on a fix for this issue.
AddedUpdateidentified#10
Before: not present
After: We are actively fixing this issue and will report back as soon this resolves. We are very sorry for the inconvenience in the meantime.
AddedUpdateresolved#11
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Critical incident: Queued jobs are taking time to process (this change)
Travis CI reported critical delays processing queued builds, affecting Linux, Windows and Mac build processing. The incident started at 2021-06-21 15:05 UTC and resolved at 2021-06-22 08:05 UTC, lasting 16 h 59 min. Travis CI identified the problem and worked on a fix but provided no cause or fix details.
Major incident

All Travis CI changes → · Travis CI profile · Markdown · JSON