Companies / GitLab / incidents

GitLab: CI Pipeline processing delays

Affected components: Background Processing

Timeline of updates

1. investigating2026-03-12 17:05 UTC

   Our team is investigating potential delays with background job processing. This could impact CI/CD job processing.

2. investigating2026-03-12 17:12 UTC

   We're seeing increased job queuing rates that are primarily affecting small CI runners. Our team is still investigating.

3. investigating2026-03-12 17:27 UTC

   No material updates at this time, our team is still investigating the issue.

4. investigating2026-03-12 17:35 UTC

   Our team is still investigating the issue.

5. investigating2026-03-12 17:59 UTC

   Our team is seeing some recovery but are still investigating the root cause. We believe that a previous Sidekiq backlog caused CI jobs to accumulate, leading to delays. More detail in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21532

6. monitoring2026-03-12 18:06 UTC

   The backlog has recovered and we are no longer seeing delays. Our investigation points to a Sidekiq backlog which lead to delays in CI job execution. Our team is currently monitoring. Additional details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21532

7. monitoring2026-03-12 18:33 UTC

   For clarification, it was noted earlier in the incident that we saw GitLab-hosted Runners were primarily impacted. However, the background job processing delay has impact on self-hosted runners as well. Our team has seen a recovery and we are still monitoring for stability.

8. resolved2026-03-12 20:07 UTC

   Our team has been monitoring and job processing has returned to normal. This incident is now resolved. Further details available in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21532

Impact, status and times are the vendor’s own, as last archived. View on status.gitlab.com (opens in new tab). GitLab profile · Recent GitLab incidents · Markdown · JSON