# Buildkite change history Filter: kind=all. Query: `?kind=notable|all|pages|pricing|trust|legal|incident|incident_edit|incident_deleted|incident_update&history=&page=N`. Company profile: [Buildkite](https://rewindex.com/company/buildkite.md) ## Changes by kind | Kind | Changes | Notable | Latest (UTC) | | --- | --- | --- | --- | | Major incident | 14 | 14 | 2026-09-30 06:30 UTC | ## Timeline | When (UTC) | Kind | Change | Label | Summary | | --- | --- | --- | --- | --- | | 2026-09-30 06:30 UTC | Major incident | [Major incident: macOS hosted agent jobs not starting](https://rewindex.com/change/i62cfc631939026.md) | outage | Buildkite reported a major outage preventing macOS hosted agent jobs from starting because of a misconfigured agent image update. Recorded start: 2026-09-30 06:30 UTC; recorded resolution: 2026-09-28 01:30 UTC; duration unknown. The vendor reported a fix and normal operation for new jobs; affected jobs still in Scheduled must be canceled and retried. | | 2026-09-22 20:07 UTC | Major incident | [Major incident: Issues with webhook ingestion, GraphQL queries](https://rewindex.com/change/i568367573ff6c0.md) | outage | Buildkite reported a major incident affecting webhook ingestion, GraphQL queries and some web requests, starting at 2026-09-22 20:07 UTC and resolved at 2026-09-22 20:50 UTC (42 min). The postmortem places customer impact between 19:09 and 19:32 UTC: 2.0% of webhook ingestion requests and 20.3% of external GraphQL requests returned HTTP 500 errors. | | 2026-09-09 19:07 UTC | Major incident | [Major incident: Buildkite service disruption](https://rewindex.com/change/i56f5367ec0a2a2.md) | outage | Buildkite’s major disruption affected Web UI, REST API, Agent API and job queues, causing failed requests, delayed scheduling and artifact-upload issues. Started 2026-09-09 19:07 UTC; resolved 2026-09-10 01:54 UTC (6 h 46 min), with full customer recovery reported since 20:28 UTC. An autoscaling feedback loop overwhelmed the Redis cluster with connections. | | 2026-09-02 14:45 UTC | Major incident | [Major incident: Delays in Hosted Agents job dispatch](https://rewindex.com/change/ife4421a78d2f13.md) | degradation | Buildkite reported a major Hosted Agents job-dispatch degradation affecting Linux (AMD64), Linux (ARM64), and MacOS from 2026-09-02 14:45 UTC to 2026-09-02 15:51 UTC, lasting 1 h 5 min. Self-hosted agents were unaffected; Buildkite worked through the delayed-job backlog and confirmed dispatch delays were resolved. | | 2026-08-25 22:55 UTC | Major incident | [Major incident: Buildkite service disruption](https://rewindex.com/change/ib9701ab3186c01.md) | outage | Buildkite reported a major site-wide outage starting 2026-08-25 22:55 UTC, resolved 2026-08-26 00:18 UTC (1 h 22 min). Internal DNS failure blocked new builds and delayed job dispatch; web, APIs and notifications were affected. Mitigations restored service; no build requests were lost. The postmortem places customer disruption at 22:47 UTC to 23:16 UTC. | | 2026-08-03 05:11 UTC | Major incident | [Major incident: Latency on Pipelines](https://rewindex.com/change/i8c132440fb7b93.md) | degradation | Buildkite reported a major Job Queue degradation starting 2026-08-03 05:11 UTC, resolved at 2026-08-03 05:36 UTC after 25 min. Scheduled builds were delayed, alongside job dispatch, pipeline uploads and incoming webhooks. A fix was fully deployed at 05:29 UTC; Buildkite said affected events would be automatically retried. | | 2026-06-18 08:05 UTC | Major incident | [Major incident: Increased latency and error rates](https://rewindex.com/change/i7966fe4522473e.md) | degradation | Buildkite reported a major degradation affecting the REST API’s builds create API for a subset of tenants, with increased latency and error rates. It started at 2026-06-18 08:05 UTC and lasted 1 h 37 min, resolving at 2026-06-18 09:43 UTC; a fix was deployed and full recovery confirmed. | | 2026-06-11 23:18 UTC | Major incident | [Major incident: Increased latency on REST and GraphQL APIs](https://rewindex.com/change/i31ae7faaa4fab9.md) | degradation | Buildkite reported a major incident involving increased REST and GraphQL API latency for all customers, starting 2026-06-11 23:18 UTC and resolved 2026-06-12 00:16 UTC (57 min). Buildkite attributed the issue to elevated REST API service load and confirmed recovery after mitigation; the agent and stacks API were unaffected. | | 2026-05-28 20:20 UTC | Major incident | [Major incident: Delayed notifications](https://rewindex.com/change/i5d667e64406048.md) | degradation | Buildkite reported a major incident delaying build and job notifications for a subset of customers, affecting email, GitHub commit status, Slack and webhooks. It started at 2026-05-28 20:20 UTC and was resolved at 2026-05-28 21:18 UTC; the computed duration was 57 min. Delayed notifications also delayed downstream CI, merge or deployment workflows for some customers. | | 2026-05-20 16:40 UTC | Major incident | [Major incident: Delayed notifications](https://rewindex.com/change/i0a7d8758f669de.md) | degradation | Buildkite reported a major incident affecting a subset of customers, with delayed email, GitHub commit status, Slack and webhook notifications; the postmortem also noted build dispatch and metrics latency. It started at 2026-05-20 16:40 UTC and resolved at 2026-05-20 17:39 UTC, lasting 59 min. Buildkite applied mitigations and monitored recovery. | | 2026-05-07 22:45 UTC | Major incident | [Major incident: Delays in job dispatch, webhook processing, and outbound webhooks](https://rewindex.com/change/i14ca98fe8f97a5.md) | degradation | Buildkite reported a major degradation delaying job dispatch, inbound and outbound webhooks, and build notifications. The incident started at 2026-05-07 22:45 UTC and resolved at 2026-05-08 00:13 UTC, lasting 1 h 28 min. All customer workloads recovered; the postmortem reported no data loss. | | 2026-05-07 08:48 UTC | Major incident | [Major incident: Jobs not starting on hosted agents and agent-stack-k8s](https://rewindex.com/change/i007ca0c579609e.md) | outage | Buildkite reported a major Agent API outage: jobs failed to start on hosted agents, agent-stack-k8s and other implementations using acquire job. The incident started at 2026-05-07 08:48 UTC and was resolved at 2026-05-07 09:37 UTC, lasting 49 minutes. Buildkite reverted a change that supplied stale environment variables to acquire job. | | 2026-05-01 02:41 UTC | Major incident | [Major incident: Buildkite service disruption](https://rewindex.com/change/i07fb9a271f2c00.md) | outage | Buildkite reported a major disruption starting 2026-05-01 02:41 UTC, lasting 49 min. The postmortem says the dashboard, Agent API, REST API and GraphQL were severely degraded for most customers between 02:29 and 02:48 UTC, preventing reliable dashboard access and causing agent communication errors. Buildkite identified a service change, announced a rollback, and subsequently reported normal service restored. | | 2026-04-08 22:26 UTC | Major incident | [Major incident: Degraded performance and increased error rates](https://rewindex.com/change/i394c4c0f41ef48.md) | degradation | Buildkite reported a major degradation starting 2026-04-08 22:26 UTC, resolved at 2026-04-08 23:12 UTC (45 min). Customers experienced website and API latency, triggered and scheduled build delays up to 12 minutes, and Agent API errors; notifications were queued and processed. Buildkite said it fixed the issue and notification latency returned to normal. | --- Canonical: https://rewindex.com/company/buildkite/changes · Markdown: https://rewindex.com/company/buildkite/changes.md · JSON: https://rewindex.com/api/v1/companies/buildkite/changes · Generated 2026-10-18T02:53:42.473Z