Changes / GitLab / CI/CD pipeline processing delays
GitLab: Major incident: CI/CD pipeline processing delays
Summary
Summary pending. The diff below shows exactly what changed.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: status.gitlab.com (opens in new tab)
Incident page: full timeline of updates →
What changed
Before: not present
After: CI/CD pipeline processing delays
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-03-09 15:59 UTC
Before: not present
After: 2026-03-09 18:15 UTC
Before: not present
After: CI/CD
Before: not present
After: We are seeing delays in CI/CD pipeline processing. The cause for this problem is under investigation. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21484 for more details.
Before: not present
After: Our team is still investigating this issue. Users might encounter some delays with processing CI/CD pipelines.
Before: not present
After: Our team is considering increasing resources to address the issue with the Sidekiq backlog impacting CI/CD pipelines. We're working on a deployment fix to address this, though we are seeing some improvement overall with queue durations.
Before: not present
After: Our team has turned off a specific worker to help improve pipeline processing. There has been significant recovery, and our team has identified this as a resource issue. We are monitoring for stability in addition to investigating the root cause and determining a permanent fix.
Before: not present
After: We are continuing to monitor but the main backlog drivers are mitigated. Users should expect normal operations. Our team is currently deploying a configuration update to further address this. We'll provide an update once it is deployed.
Before: not present
After: The configuration change has been deployed and we have seen stability with Sidekiq and pipeline processing. This incident is now resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: CI/CD pipeline processing delays (this change) | Major incident |