# GitLab: Major incident: CI/CD pipelines intermittently stuck on running status

**Summary:** _summary pending_
- Kind: Major incident · Label: major (vendor)
- Changed: 2026-03-05 21:25 UTC · Detected: 2026-10-18 05:08 UTC
- [Source: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/69a9f4b650f2320a21a9175a](https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/69a9f4b650f2320a21a9175a)
- [Company: GitLab](https://rewindex.com/company/gitlab.md)
- Versions: before (none: first archived) → after incident\_version:188/69a9f4b650f2320a21a9175a@8e7339ae74e2
- Change 1 of 1 for this incident

Incident page (full timeline): [CI/CD pipelines intermittently stuck on running status](https://rewindex.com/company/gitlab/incidents/69a9f4b650f2320a21a9175a.md)
## What changed

**1. Added: Title**

```diff
+ CI/CD pipelines intermittently stuck on running status
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-03-05 21:25 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-03-06 00:04 UTC
```

**6. Added: Affected components**

```diff
+ CI/CD
```

**7. Added: Update** (investigating) · 2026-03-05 21:25 UTC

```diff
+ Some users are experiencing CI/CD pipelines where jobs and pipelines are stuck in a “running” state, even though earlier-stage jobs have completed successfully. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

**8. Added: Update** (investigating) · 2026-03-05 21:40 UTC

```diff
+ No material change since the last update. The incident remains under active investigation. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

**9. Added: Update** (investigating) · 2026-03-05 21:55 UTC

```diff
+ We’re still actively investigating and are currently collating data from affected pipelines to better understand the patterns and root cause. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

**10. Added: Update** (investigating) · 2026-03-05 22:12 UTC

```diff
+ We are currently reviewing logs from our production infrastructure and specifically the status and behavior of the pipeline workers for potential causes. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

**11. Added: Update** (investigating) · 2026-03-05 22:25 UTC

```diff
+ We’ve identified a potential issue with the job deferral mechanism that may be affecting pipeline transitions between stages. For details, please see https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462.
```

**12. Added: Update** (identified) · 2026-03-05 23:19 UTC

```diff
+ We have identified the cause of this incident as the recurrence of an existing defect in the pipeline job deferral mechanism. In this case, it was triggered by a period of temporary high pipeline concurrency. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

**13. Added: Update** (monitoring) · 2026-03-05 23:27 UTC

```diff
+ At this time, we are confident that current pipelines are not being affected by this behavior. We are continuing to monitor new reports closely until the top of the hour to ensure there is no ongoing impact. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

**14. Added: Update** (resolved) · 2026-03-06 00:04 UTC

```diff
+ As there is no current impact, we are setting this incide to Resolved. We have identified the problematic behavior behind it and are working on a long-term fix. However, we know this defect can reoccur under certain high‑concurrency conditions. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21462 for details.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-03-05 21:25 UTC | Major incident: CI/CD pipelines intermittently stuck on running status (this change) | major |

All GitLab changes: [timeline](https://rewindex.com/company/gitlab/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ia344ad2eb09afe · Markdown: https://rewindex.com/change/ia344ad2eb09afe.md · JSON: https://rewindex.com/api/v1/changes/ia344ad2eb09afe · Generated 2026-10-18T10:23:40.300Z