Companies / GitLab / incidents

GitLab: Degraded performance for pipelines on gitlab.com

Impact: criticalStatus: resolvedStarted Resolved (30 h)20 updatesSource: status.gitlab.com (opens in new tab)

Affected components: Background ProcessingCI/CDCI/CD - Hosted runners for GitLab community contributionsCI/CD - Hosted runners on LinuxCI/CD - Hosted runners on macOSCI/CD - Hosted runners on Windows

Timeline of updates

  1. investigating

    We have received reports of delayed or stuck pipelines, and jobs waiting longer than expected to execute. We are investigating. See issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  2. investigating

    Investigation is still underway. An MR to increase resource limits to mitigate impact is currently awaiting merge. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  3. investigating

    Resource limits have been increased and we are monitoring the impact. Investigation into the root cause is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  4. investigating

    The resource limits increase did not sufficiently mitigate impact. Investigation into cause and mitigation options is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  5. investigating

    Mitigation action has been taken and new pipelines can be triggered. Pipelines are being processed, but delays are still expected for up to 2 hours. Jobs may still be stuck in pending state during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  6. identified

    We have identified the cause and taken action to mitigate re-occurance. We are seeing recovery on CI workloads and it should return to normal within the next hour. Some delays are still expected during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  7. monitoring

    CI workloads are recovering as expected and should return to normal within the next hour. Some pipelines may still be delayed during this time. We are currently monitoring. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  8. identified

    We are continuing to see delays in CI job processing and are actively investigating mitigation steps. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  9. investigating

    In addition to CI job processing delays, users may also see "Failed to retrieve CI/CD variables from cache" when running new pipelines manually via the UI. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  10. investigating

    Issues with Hosted Runner CI job queue delays and cache errors related to CI/CD variables are still under investigation. Self-hosted Runners are not currently affected. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  11. investigating

    No material update available at this time. Investigation into CI job delays on Hosted Runners and CI variable retrieval is ongoing. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  12. investigating

    We have observed some improvement in queue times but are still investigating CI job delays on Hosted Runners and CI variable retrieval. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  13. investigating

    We have observed that jobs on Hosted Runners are processing without delay and are able to retrieve cached CI/CD variables. We are currently awaiting deployment of a fix to ensure the issue remains mitigated. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  14. investigating

    Hosted Runners continue to process without delay and are able to retrieve cached CI/CD variables. However some background processing is currently impacted and users may observe environments/review apps URLs not updating. We continue to wait for the fix to reach production. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  15. investigating

    Background processing remains degraded, affecting Environments & Review Apps. Hosted Runners and CI jobs are healthy. A revert MR is queued for production, with more changes needed to address background processing. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  16. investigating

    Background processing remains degraded, however the revert MR has been merged a short time ago. We are monitoring performance to see if the MR results in performance returning to optimal levels. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190

  17. monitoring

    We've disabled the worker responsible for the issues. We've merged another fix for that specific worker and are in the process of deploying it to production before re-enabling it. The next update will be posted once that is completed.

  18. identified

    Background processing remains degraded, affecting Environments and Review Apps, a fix is awaiting deployment. Hosted Runners and CI jobs are healthy. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  19. monitoring

    We have applied a fix and are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.

  20. resolved

    This issue is now resolved. The CI job queue on Hosted Runners and Environments/Review Apps have remained stable during our monitoring period.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on status.gitlab.com (opens in new tab). GitLab profile · Recent GitLab incidents · Markdown · JSON