Buildkite change history

Buildkite reported a major outage preventing macOS hosted agent jobs from starting because of a misconfigured agent image update. Recorded start: 2026-09-30 06:30 UTC; recorded resolution: 2026-09-28 01:30 UTC; duration unknown. The vendor reported a fix and normal operation for new jobs; affected jobs still in Scheduled must be canceled and retried.

Buildkite reported a major incident affecting webhook ingestion, GraphQL queries and some web requests, starting at 2026-09-22 20:07 UTC and resolved at 2026-09-22 20:50 UTC (42 min). The postmortem places customer impact between 19:09 and 19:32 UTC: 2.0% of webhook ingestion requests and 20.3% of external GraphQL requests returned HTTP 500 errors.

Buildkite’s major disruption affected Web UI, REST API, Agent API and job queues, causing failed requests, delayed scheduling and artifact-upload issues. Started 2026-09-09 19:07 UTC; resolved 2026-09-10 01:54 UTC (6 h 46 min), with full customer recovery reported since 20:28 UTC. An autoscaling feedback loop overwhelmed the Redis cluster with connections.

Buildkite reported a major Hosted Agents job-dispatch degradation affecting Linux (AMD64), Linux (ARM64), and MacOS from 2026-09-02 14:45 UTC to 2026-09-02 15:51 UTC, lasting 1 h 5 min. Self-hosted agents were unaffected; Buildkite worked through the delayed-job backlog and confirmed dispatch delays were resolved.

Buildkite reported a major site-wide outage starting 2026-08-25 22:55 UTC, resolved 2026-08-26 00:18 UTC (1 h 22 min). Internal DNS failure blocked new builds and delayed job dispatch; web, APIs and notifications were affected. Mitigations restored service; no build requests were lost. The postmortem places customer disruption at 22:47 UTC to 23:16 UTC.

Buildkite reported a major Job Queue degradation starting 2026-08-03 05:11 UTC, resolved at 2026-08-03 05:36 UTC after 25 min. Scheduled builds were delayed, alongside job dispatch, pipeline uploads and incoming webhooks. A fix was fully deployed at 05:29 UTC; Buildkite said affected events would be automatically retried.

Buildkite reported a major degradation affecting the REST API’s builds create API for a subset of tenants, with increased latency and error rates. It started at 2026-06-18 08:05 UTC and lasted 1 h 37 min, resolving at 2026-06-18 09:43 UTC; a fix was deployed and full recovery confirmed.

Buildkite reported a major incident involving increased REST and GraphQL API latency for all customers, starting 2026-06-11 23:18 UTC and resolved 2026-06-12 00:16 UTC (57 min). Buildkite attributed the issue to elevated REST API service load and confirmed recovery after mitigation; the agent and stacks API were unaffected.

Buildkite reported a major incident delaying build and job notifications for a subset of customers, affecting email, GitHub commit status, Slack and webhooks. It started at 2026-05-28 20:20 UTC and was resolved at 2026-05-28 21:18 UTC; the computed duration was 57 min. Delayed notifications also delayed downstream CI, merge or deployment workflows for some customers.

Buildkite reported a major incident affecting a subset of customers, with delayed email, GitHub commit status, Slack and webhook notifications; the postmortem also noted build dispatch and metrics latency. It started at 2026-05-20 16:40 UTC and resolved at 2026-05-20 17:39 UTC, lasting 59 min. Buildkite applied mitigations and monitored recovery.

Buildkite reported a major degradation delaying job dispatch, inbound and outbound webhooks, and build notifications. The incident started at 2026-05-07 22:45 UTC and resolved at 2026-05-08 00:13 UTC, lasting 1 h 28 min. All customer workloads recovered; the postmortem reported no data loss.

Buildkite reported a major Agent API outage: jobs failed to start on hosted agents, agent-stack-k8s and other implementations using acquire job. The incident started at 2026-05-07 08:48 UTC and was resolved at 2026-05-07 09:37 UTC, lasting 49 minutes. Buildkite reverted a change that supplied stale environment variables to acquire job.

Buildkite reported a major disruption starting 2026-05-01 02:41 UTC, lasting 49 min. The postmortem says the dashboard, Agent API, REST API and GraphQL were severely degraded for most customers between 02:29 and 02:48 UTC, preventing reliable dashboard access and causing agent communication errors. Buildkite identified a service change, announced a rollback, and subsequently reported normal service restored.

Buildkite reported a major degradation starting 2026-04-08 22:26 UTC, resolved at 2026-04-08 23:12 UTC (45 min). Customers experienced website and API latency, triggered and scheduled build delays up to 12 minutes, and Agent API errors; notifications were queued and processed. Buildkite said it fixed the issue and notification latency returned to normal.