Changes / GitLab / CI/CD pipelines intermittently stuck on running status
GitLab: Major incident: CI/CD pipelines intermittently stuck on running status
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: full timeline of updates →
What changed
Before: not present
After: CI/CD pipelines intermittently stuck on running status
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-03-05 21:25 UTC
Before: not present
After: 2026-03-06 00:04 UTC
Before: not present
After: CI/CD
Before: not present
After: Some users are experiencing CI/CD pipelines where jobs and pipelines are stuck in a “running” state, even though earlier-stage jobs have completed successfully. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
Before: not present
After: No material change since the last update. The incident remains under active investigation. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
Before: not present
After: We’re still actively investigating and are currently collating data from affected pipelines to better understand the patterns and root cause. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
Before: not present
After: We are currently reviewing logs from our production infrastructure and specifically the status and behavior of the pipeline workers for potential causes. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
Before: not present
After: We’ve identified a potential issue with the job deferral mechanism that may be affecting pipeline transitions between stages. For details, please see https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462.
Before: not present
After: We have identified the cause of this incident as the recurrence of an existing defect in the pipeline job deferral mechanism. In this case, it was triggered by a period of temporary high pipeline concurrency. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
Before: not present
After: At this time, we are confident that current pipelines are not being affected by this behavior. We are continuing to monitor new reports closely until the top of the hour to ensure there is no ongoing impact. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
Before: not present
After: As there is no current impact, we are setting this incide to Resolved. We have identified the problematic behavior behind it and are working on a long-term fix. However, we know this defect can reoccur under certain high‑concurrency conditions. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: CI/CD pipelines intermittently stuck on running status (this change) | Major incident |