Search

Companies / GitLab / incidents

GitLab: 500 Errors due to failures in Gitaly service nodes

Impact: majorStatus: resolvedStarted Resolved (19 h)13 updatesSource: status.gitlab.com (opens in new tab)

Affected components: CanaryGit OperationsWebsite

Timeline of updates

  1. identified

    We see a 1.5% degraded state on two of our Gitaly nodes. Some customer may experience delays while contributing to their projects. We have identified the problem and we are working through steps to resolve this. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  2. identified

    Partial service disruption due to this issue.

  3. monitoring

    We have recovered and are still monitoring. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  4. investigating

    We have seen an unexpected increase in 500 errors on GitLab.com pages, often related to Merge Requests. We are investigating the root cause of these issues. Further details in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  5. investigating

    We are continuing to investigate the 500 errors on GitLab.com. We are investigating the root cause of the incident. We are preparing to apply mitigations to reduce error rates. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  6. investigating

    We have rebooted the VMs that were generating 500 errors. Once the VMs have rebooted we will validate that Gitaly is responding normally and work to identify the root cause of these issues to prevent re-occurrence. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  7. monitoring

    The nodes have been rebooted and Gitaly appears to be working as expected. We are moving this incident to Monitoring. Our team is working to identify the root cause of these errors while we monitor the node performance. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  8. monitoring

    Gitaly appears to be working normally at this time. We are continuing to monitor the issue and have mitigation steps should the issue reoccur. We do not anticipate further updates unless we start seeing degraded Gitaly performance again. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  9. investigating

    We're observing elevated 500 error rate on GitLab.com again and we're investigating the root cause. See detail issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  10. identified

    We've identified the problem related to git activities. We're looking into solution. See issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  11. identified

    We're still exploring the solution for the git problem. See details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  12. monitoring

    Restarting specific nodes improved the situation. We'll keep monitoring the situation for now. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  13. resolved

    Restarting specific nodes improved the situation. We'll mark the incident as resolve while we keep monitoring the situation. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on status.gitlab.com (opens in new tab). GitLab profile · Recent GitLab incidents · Markdown · JSON