# GitLab: Degraded performance for pipelines on gitlab.com - Impact (vendor’s label): critical - Status: resolved - Started: 2026-02-03 11:36 UTC · Resolved: 2026-02-04 17:40 UTC · Duration: 30 h - Updates: 20 · Affected components: Background Processing, CI/CD, CI/CD - Hosted runners for GitLab community contributions, CI/CD - Hosted runners on Linux, CI/CD - Hosted runners on macOS, CI/CD - Hosted runners on Windows - [Vendor status page: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6981ddbdb85f11053f16f029](https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6981ddbdb85f11053f16f029) - [Company: GitLab](https://rewindex.com/company/gitlab.md) ## Timeline of updates - **investigating** 2026-02-03 11:36 UTC: We have received reports of delayed or stuck pipelines, and jobs waiting longer than expected to execute. We are investigating. See issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **investigating** 2026-02-03 12:08 UTC: Investigation is still underway. An MR to increase resource limits to mitigate impact is currently awaiting merge. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **investigating** 2026-02-03 12:39 UTC: Resource limits have been increased and we are monitoring the impact. Investigation into the root cause is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **investigating** 2026-02-03 13:51 UTC: The resource limits increase did not sufficiently mitigate impact. Investigation into cause and mitigation options is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **investigating** 2026-02-03 14:28 UTC: Mitigation action has been taken and new pipelines can be triggered. Pipelines are being processed, but delays are still expected for up to 2 hours. Jobs may still be stuck in pending state during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **identified** 2026-02-03 15:19 UTC: We have identified the cause and taken action to mitigate re-occurance. We are seeing recovery on CI workloads and it should return to normal within the next hour. Some delays are still expected during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **monitoring** 2026-02-03 15:31 UTC: CI workloads are recovering as expected and should return to normal within the next hour. Some pipelines may still be delayed during this time. We are currently monitoring. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **identified** 2026-02-03 17:21 UTC: We are continuing to see delays in CI job processing and are actively investigating mitigation steps. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-03 17:44 UTC: In addition to CI job processing delays, users may also see "Failed to retrieve CI/CD variables from cache" when running new pipelines manually via the UI. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-03 18:09 UTC: Issues with Hosted Runner CI job queue delays and cache errors related to CI/CD variables are still under investigation. Self-hosted Runners are not currently affected. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-03 18:43 UTC: No material update available at this time. Investigation into CI job delays on Hosted Runners and CI variable retrieval is ongoing. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-03 20:33 UTC: We have observed some improvement in queue times but are still investigating CI job delays on Hosted Runners and CI variable retrieval. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-03 23:29 UTC: We have observed that jobs on Hosted Runners are processing without delay and are able to retrieve cached CI/CD variables. We are currently awaiting deployment of a fix to ensure the issue remains mitigated. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-04 01:29 UTC: Hosted Runners continue to process without delay and are able to retrieve cached CI/CD variables. However some background processing is currently impacted and users may observe environments/review apps URLs not updating. We continue to wait for the fix to reach production. Details in gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **investigating** 2026-02-04 03:29 UTC: Background processing remains degraded, affecting Environments & Review Apps. Hosted Runners and CI jobs are healthy. A revert MR is queued for production, with more changes needed to address background processing. See: gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **investigating** 2026-02-04 06:31 UTC: Background processing remains degraded, however the revert MR has been merged a short time ago. We are monitoring performance to see if the MR results in performance returning to optimal levels. See: gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190 - **monitoring** 2026-02-04 09:56 UTC: We've disabled the worker responsible for the issues. We've merged another fix for that specific worker and are in the process of deploying it to production before re-enabling it. The next update will be posted once that is completed. - **identified** 2026-02-04 10:33 UTC: Background processing remains degraded, affecting Environments and Review Apps, a fix is awaiting deployment. Hosted Runners and CI jobs are healthy. Details in gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **monitoring** 2026-02-04 14:25 UTC: We have applied a fix and are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/work\_items/21190. - **resolved** 2026-02-04 17:40 UTC: This issue is now resolved. The CI job queue on Hosted Runners and Environments/Review Apps have remained stable during our monitoring period. ## Rewindex changes for this incident - 2026-02-03 11:36 UTC: [Critical incident: Degraded performance for pipelines on gitlab.com](https://rewindex.com/change/i82282a08b53009.md) --- Canonical: https://rewindex.com/company/gitlab/incidents/6981ddbdb85f11053f16f029 · Markdown: https://rewindex.com/company/gitlab/incidents/6981ddbdb85f11053f16f029.md · JSON: https://rewindex.com/api/v1/companies/gitlab/incidents/6981ddbdb85f11053f16f029 · Generated 2026-10-18T10:26:03.061Z