Changes / GitLab / 500 Errors due to failures in Gitaly service nodes
GitLab: Major incident: 500 Errors due to failures in Gitaly service nodes
Summary
Summary pending. The diff below shows exactly what changed.
Major incidentLabel: outage (model (Jev), 99%)Changed Detected Source: status.gitlab.com (opens in new tab)
Incident page: full timeline of updates →
What changed
Before: not present
After: 500 Errors due to failures in Gitaly service nodes
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-02-17 11:54 UTC
Before: not present
After: 2026-02-18 07:00 UTC
Before: not present
After: Canary, Git Operations, Website
Before: not present
After: We see a 1.5% degraded state on two of our Gitaly nodes. Some customer may experience delays while contributing to their projects. We have identified the problem and we are working through steps to resolve this. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: Partial service disruption due to this issue.
Before: not present
After: We have recovered and are still monitoring. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: We have seen an unexpected increase in 500 errors on GitLab.com pages, often related to Merge Requests. We are investigating the root cause of these issues. Further details in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: We are continuing to investigate the 500 errors on GitLab.com. We are investigating the root cause of the incident. We are preparing to apply mitigations to reduce error rates. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310
Before: not present
After: We have rebooted the VMs that were generating 500 errors. Once the VMs have rebooted we will validate that Gitaly is responding normally and work to identify the root cause of these issues to prevent re-occurrence. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310
Before: not present
After: The nodes have been rebooted and Gitaly appears to be working as expected. We are moving this incident to Monitoring. Our team is working to identify the root cause of these errors while we monitor the node performance. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310
Before: not present
After: Gitaly appears to be working normally at this time. We are continuing to monitor the issue and have mitigation steps should the issue reoccur. We do not anticipate further updates unless we start seeing degraded Gitaly performance again. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310
Before: not present
After: We're observing elevated 500 error rate on GitLab.com again and we're investigating the root cause. See detail issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: We've identified the problem related to git activities. We're looking into solution. See issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: We're still exploring the solution for the git problem. See details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: Restarting specific nodes improved the situation. We'll keep monitoring the situation for now. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
Before: not present
After: Restarting specific nodes improved the situation. We'll mark the incident as resolve while we keep monitoring the situation. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: 500 Errors due to failures in Gitaly service nodes (this change) | Major incident |