Changes / Buildkite / Degraded performance and increased error rates
Buildkite: Major incident: Degraded performance and increased error rates
Summary
Buildkite reported a major degradation starting 2026-04-08 22:26 UTC, resolved at 2026-04-08 23:12 UTC (45 min). Customers experienced website and API latency, triggered and scheduled build delays up to 12 minutes, and Agent API errors; notifications were queued and processed. Buildkite said it fixed the issue and notification latency returned to normal.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 97%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Degraded performance and increased error rates
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-04-08 22:26 UTC
Before: not present
After: 2026-04-08 23:12 UTC
Before: not present
After: Agent API, Email Notifications, GitHub Commit Status Notifications, Notifications - Email Notifications, Notifications - GitHub Commit Status Notifications, Notifications - Slack Notifications, Notifications - Webhook Notifications, Slack Notifications, Webhook Notifications
Before: not present
After: We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Before: not present
After: We have identified and fixed the issue. We are monitoring and seeing signs of improvement.
Before: not present
After: We experienced an issue that caused a brief increase in errors for the Agent API. This also impacted latency for notifications. All notifications were stored in a queue and processed. Latency is now back to normal.
Before: not present
After: ## Service Impact
On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and GraphQL APIs, as well as latency creating triggered and scheduled builds of up to 12 minutes.
A portion of customers also experienced increased latency and error rates for the Agent API, with the impact not being evenly distributed, causing some customers to experience a higher impact than others. Customers impacted experienced p99 latency of more than 1 second and error rates of 0.5%.
Between 22:10 to 23:03 UTC a majority of customers experienced notification latency of between 5 and 33 minutes, which includes Github commit statuses and webhook delivery, as well as delays in processing incoming webhooks of up to 40 seconds.
## Incident Summary
Our engineers noticed an increase in exceptions and shortly afterwards received an alert at 22:16 UTC for high CPU utilisation on a single node in a Redis cluster. Upon investigation we found a rate limiter was hot spotting on a single node within the cluster. Upon removing this limit at 22:33 UTC the impact of high Redis CPU utilization was mitigated, with CPU utilization falling to 5% for the affected node where it had been at 50% prior to the incident. Further investigations also revealed high load on a replica database was also contributing to high latency, which recovered at 22:37 UTC after a third replica was added.
The rate limit which was responsible for high load on Redis had caused hot spotting on a single key because it was applied across all organizations. This limit was introduced in response to a previous incident but hadn’t been required since our work to horizontally shard our Pipelines database had distributed the load and enabled higher scalability in early 2024.
## Changes we've made
These are the changes we’ve made in response to this incident:
* Removed the global rate limit that was contributing to a significant proportion of load
* Increased the number of read replicas for our customer information database to 3 and increased the instance size
* We’ve reviewed our remaining rate limits and have confirmed that no other rate limits apply globally to all shards.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Degraded performance and increased error rates (this change) Buildkite reported a major degradation starting 2026-04-08 22:26 UTC, resolved at 2026-04-08 23:12 UTC (45 min). Customers experienced website and API latency, triggered and scheduled build delays up to 12 minutes, and Agent API errors; notifications were queued and processed. Buildkite said it fixed the issue and notification latency returned to normal. | Major incident |
All Buildkite changes → · Buildkite profile · Markdown · JSON