Changes / GitLab / Delay in background job processing
GitLab: Major incident: Delay in background job processing
Summary
Summary pending. The diff below shows exactly what changed.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: status.gitlab.com (opens in new tab)
Incident page: full timeline of updates →
What changed
Before: not present
After: Delay in background job processing
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-03-17 15:51 UTC
Before: not present
After: 2026-03-20 15:38 UTC
Before: not present
After: Background Processing
Before: not present
After: We are investigating delay in background job processing. We are working on further improvements, more details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: Improvements to help address this issue are currently under review. We are still observing a delay in background job processing. Users may still see delays in CI job status updates. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: Background job processing has recovered as peak load has passed. Additional changes are under review to address this going forward. This incident will remain in monitoring as we deploy these changes and observe metrics through tomorrow's peak. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: We are investigating delay in background job processing. After a monitoring period that followed yesterday fixes, and expanding infra resources, we are investigating further improvements, more details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: Background job processing has recovered. Additional changes have been implemented which helped to reduce impact during peak load. We will continue to monitor this as we move through peak load times tomorrow. Details: http://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: The team has continued to make improvements since yesterday's update to further reduce impact during peak load. We are actively monitoring the situation. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: We are currently investigating performance degradation and our team is currently actively working on a mitigation. More details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: We observed an increased delay in background job processing and CI job queue times, both of which have now recovered. We're closely monitoring the situation as we work on additional improvements. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: The team has continued to make improvements to the underlying infrastructure to prevent further impact. We have not seen any recent occurrences of background job delays. We will keep this incident open through peak load times over the next 20 hours to monitor performance metrics. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Before: not present
After: The combination of infrastructure and application improvements made has sustained stability during peak hours yesterday. This incident has been moved to resolved following our monitoring. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delay in background job processing (this change) | Major incident |