Changes / GitLab / Degraded performance for pipelines on gitlab.com
GitLab: Critical incident: Degraded performance for pipelines on gitlab.com
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: full timeline of updates →
What changed
Before: not present
After: Degraded performance for pipelines on gitlab.com
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2026-02-03 11:36 UTC
Before: not present
After: 2026-02-04 17:40 UTC
Before: not present
After: Background Processing, CI/CD, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners on Windows
Before: not present
After: We have received reports of delayed or stuck pipelines, and jobs waiting longer than expected to execute. We are investigating. See issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: Investigation is still underway. An MR to increase resource limits to mitigate impact is currently awaiting merge. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: Resource limits have been increased and we are monitoring the impact. Investigation into the root cause is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: The resource limits increase did not sufficiently mitigate impact. Investigation into cause and mitigation options is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: Mitigation action has been taken and new pipelines can be triggered. Pipelines are being processed, but delays are still expected for up to 2 hours. Jobs may still be stuck in pending state during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: We have identified the cause and taken action to mitigate re-occurance. We are seeing recovery on CI workloads and it should return to normal within the next hour. Some delays are still expected during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: CI workloads are recovering as expected and should return to normal within the next hour. Some pipelines may still be delayed during this time. We are currently monitoring. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: We are continuing to see delays in CI job processing and are actively investigating mitigation steps. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: In addition to CI job processing delays, users may also see "Failed to retrieve CI/CD variables from cache" when running new pipelines manually via the UI. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: Issues with Hosted Runner CI job queue delays and cache errors related to CI/CD variables are still under investigation. Self-hosted Runners are not currently affected. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: No material update available at this time. Investigation into CI job delays on Hosted Runners and CI variable retrieval is ongoing. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: We have observed some improvement in queue times but are still investigating CI job delays on Hosted Runners and CI variable retrieval. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: We have observed that jobs on Hosted Runners are processing without delay and are able to retrieve cached CI/CD variables. We are currently awaiting deployment of a fix to ensure the issue remains mitigated. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: Hosted Runners continue to process without delay and are able to retrieve cached CI/CD variables. However some background processing is currently impacted and users may observe environments/review apps URLs not updating. We continue to wait for the fix to reach production. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: Background processing remains degraded, affecting Environments & Review Apps. Hosted Runners and CI jobs are healthy. A revert MR is queued for production, with more changes needed to address background processing. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: Background processing remains degraded, however the revert MR has been merged a short time ago. We are monitoring performance to see if the MR results in performance returning to optimal levels. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Before: not present
After: We've disabled the worker responsible for the issues. We've merged another fix for that specific worker and are in the process of deploying it to production before re-enabling it. The next update will be posted once that is completed.
Before: not present
After: Background processing remains degraded, affecting Environments and Review Apps, a fix is awaiting deployment. Hosted Runners and CI jobs are healthy. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: We have applied a fix and are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Before: not present
After: This issue is now resolved. The CI job queue on Hosted Runners and Environments/Review Apps have remained stable during our monitoring period.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Degraded performance for pipelines on gitlab.com (this change) | Major incident |