Companies / GitLab / incidents

GitLab: Delay in background job processing

Affected components: Background Processing

Timeline of updates

1. investigating2026-03-17 15:51 UTC

   We are investigating delay in background job processing. We are working on further improvements, more details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

2. identified2026-03-17 18:12 UTC

   Improvements to help address this issue are currently under review. We are still observing a delay in background job processing. Users may still see delays in CI job status updates. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

3. monitoring2026-03-17 20:17 UTC

   Background job processing has recovered as peak load has passed. Additional changes are under review to address this going forward. This incident will remain in monitoring as we deploy these changes and observe metrics through tomorrow's peak. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

4. investigating2026-03-18 15:13 UTC

   We are investigating delay in background job processing. After a monitoring period that followed yesterday fixes, and expanding infra resources, we are investigating further improvements, more details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

5. monitoring2026-03-18 17:56 UTC

   Background job processing has recovered. Additional changes have been implemented which helped to reduce impact during peak load. We will continue to monitor this as we move through peak load times tomorrow. Details: http://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

6. monitoring2026-03-19 09:41 UTC

   The team has continued to make improvements since yesterday's update to further reduce impact during peak load. We are actively monitoring the situation. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

7. investigating2026-03-19 15:36 UTC

   We are currently investigating performance degradation and our team is currently actively working on a mitigation. More details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

8. monitoring2026-03-19 17:11 UTC

   We observed an increased delay in background job processing and CI job queue times, both of which have now recovered. We're closely monitoring the situation as we work on additional improvements. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

9. monitoring2026-03-20 01:00 UTC

   The team has continued to make improvements to the underlying infrastructure to prevent further impact. We have not seen any recent occurrences of background job delays. We will keep this incident open through peak load times over the next 20 hours to monitor performance metrics. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

10. resolved2026-03-20 15:38 UTC

    The combination of infrastructure and application improvements made has sustained stability during peak hours yesterday. This incident has been moved to resolved following our monitoring. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

Rewindex changes for this incident

• Major incident: Delay in background job processing 2026-03-17 15:51 UTC

Impact, status and times are the vendor’s own, as last archived. View on status.gitlab.com (opens in new tab). GitLab profile · Recent GitLab incidents · Markdown · JSON