# Buildkite: Major incident: Degraded performance and increased error rates

**Summary:** Buildkite reported a major degradation starting 2026-04-08 22:26 UTC, resolved at 2026-04-08 23:12 UTC (45 min). Customers experienced website and API latency, triggered and scheduled build delays up to 12 minutes, and Agent API errors; notifications were queued and processed. Buildkite said it fixed the issue and notification latency returned to normal. _(machine-written, sol)_
- Kind: Major incident · Label: degradation (model (Jev))
- Changed: 2026-04-08 22:26 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/dz9t96qfqpbh](https://stspg.io/dz9t96qfqpbh)
- [Company: Buildkite](https://rewindex.com/company/buildkite.md)
- Versions: before (none: first archived) → after incident\_version:178/j7ngpbpgnw66@a931e447ca45
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Degraded performance and increased error rates
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2026-04-08 22:26 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-04-08 23:12 UTC
```

**6. Added: Affected components**

```diff
+ Agent API, Email Notifications, GitHub Commit Status Notifications, Notifications - Email Notifications, Notifications - GitHub Commit Status Notifications, Notifications - Slack Notifications, Notifications - Webhook Notifications, Slack Notifications, Webhook Notifications
```

**7. Added: Update** (investigating) · 2026-04-08 22:26 UTC

```diff
+ We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
```

**8. Added: Update** (monitoring) · 2026-04-08 22:40 UTC

```diff
+ We have identified and fixed the issue. We are monitoring and seeing signs of improvement.
```

**9. Added: Update** (resolved) · 2026-04-08 23:12 UTC

```diff
+ We experienced an issue that caused a brief increase in errors for the Agent API. This also impacted latency for notifications. All notifications were stored in a queue and processed. Latency is now back to normal.
```

**10. Added: Update** (postmortem) · 2026-04-21 06:04 UTC

```diff
+ ## Service Impact On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and GraphQL APIs, as well as latency creating triggered and scheduled builds of up to 12 minutes. A portion of customers also experienced increased latency and error rates for the Agent API, with the impact not being evenly distributed, causing some customers to experience a higher impact than others. Customers impacted experienced p99 latency of more than 1 second and error rates of 0.5%. Between 22:10 to 23:03 UTC a majority of customers experienced notification latency of between 5 and 33 minutes, which includes Github commit statuses and webhook delivery, as well as delays in processing incoming webhooks of up to 40 seconds. ## Incident Summary Our engineers noticed an increase in exceptions and shortly afterwards received an alert at 22:16 UTC for high CPU utilisation on a single node in a Redis cluster. Upon investigation we found a rate limiter was hot spotting on a single node within the cluster. Upon removing this limit at 22:33 UTC the impact of high Redis CPU utilization was mitigated, with CPU utilization falling to 5% for the affected node where it had been at 50% prior to the incident. Further investigations also revealed high load on a replica database was also contributing to high latency, which recovered at 22:37 UTC after a third replica was added. The rate limit which was responsible for high load on Redis had caused hot spotting on a single key because it was applied across all organizations. This limit was introduced in response to a previous incident but hadn’t been required since our work to horizontally shard our Pipelines database had distributed the load and enabled higher scalability in early 2024. ## Changes we've made These are the changes we’ve made in response to this incident: * Removed the global rate limit that was contributing to a significant proportion of load * Increased the number of read replicas for our customer information database to 3 and increased the instance size * We’ve reviewed our remaining rate limits and have confirmed that no other rate limits apply globally to all shards.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-04-08 22:26 UTC | Major incident: Degraded performance and increased error rates (this change) | degradation |

All Buildkite changes: [timeline](https://rewindex.com/company/buildkite/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i394c4c0f41ef48 · Markdown: https://rewindex.com/change/i394c4c0f41ef48.md · JSON: https://rewindex.com/api/v1/changes/i394c4c0f41ef48 · Generated 2026-10-18T03:55:48.091Z