Companies / Buildkite / incidents
Buildkite: Degraded performance and increased error rates
Buildkite reported a major degradation starting 2026-04-08 22:26 UTC, resolved at 2026-04-08 23:12 UTC (45 min). Customers experienced website and API latency, triggered and scheduled build delays up to 12 minutes, and Agent API errors; notifications were queued and processed. Buildkite said it fixed the issue and notification latency returned to normal.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Agent APIGitHub Commit Status NotificationsNotifications - GitHub Commit Status NotificationsNotifications - Webhook NotificationsWebhook NotificationsNotifications - Slack NotificationsSlack NotificationsEmail NotificationsNotifications - Email Notifications
Timeline of updates
- investigating
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
- monitoring
We have identified and fixed the issue. We are monitoring and seeing signs of improvement.
- resolved
We experienced an issue that caused a brief increase in errors for the Agent API. This also impacted latency for notifications. All notifications were stored in a queue and processed. Latency is now back to normal.
- postmortem
## Service Impact On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and GraphQL APIs, as well as latency creating triggered and scheduled builds of up to 12 minutes. A portion of customers also experienced increased latency and error rates for the Agent API, with the impact not being evenly distributed, causing some customers to experience a higher impact than others. Customers impacted experienced p99 latency of more than 1 second and error rates of 0.5%. Between 22:10 to 23:03 UTC a majority of customers experienced notification latency of between 5 and 33 minutes, which includes Github commit statuses and webhook delivery, as well as delays in processing incoming webhooks of up to 40 seconds. ## Incident Summary Our engineers noticed an increase in exceptions and shortly afterwards received an alert at 22:16 UTC for high CPU utilisation on a single node in a Redis cluster. Upon investigation we found a rate limiter was hot spotting on a single node within the cluster. Upon removing this limit at 22:33 UTC the impact of high Redis CPU utilization was mitigated, with CPU utilization falling to 5% for the affected node where it had been at 50% prior to the incident. Further investigations also revealed high load on a replica database was also contributing to high latency, which recovered at 22:37 UTC after a third replica was added. The rate limit which was responsible for high load on Redis had caused hot spotting on a single key because it was applied across all organizations. This limit was introduced in response to a previous incident but hadn’t been required since our work to horizontally shard our Pipelines database had distributed the load and enabled higher scalability in early 2024. ## Changes we've made These are the changes we’ve made in response to this incident: * Removed the global rate limit that was contributing to a significant proportion of load * Increased the number of read replicas for our customer information database to 3 and increased the instance size * We’ve reviewed our remaining rate limits and have confirmed that no other rate limits apply globally to all shards.
Rewindex changes for this incident
- Major incident: Degraded performance and increased error rates 2026-04-08 22:26 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Buildkite profile · Recent Buildkite incidents · Markdown · JSON