Changes / Buildkite / Increased latency and error rates
Buildkite: Major incident: Increased latency and error rates
Summary
Buildkite reported a major degradation affecting the REST API’s builds create API for a subset of tenants, with increased latency and error rates. It started at 2026-06-18 08:05 UTC and lasted 1 h 37 min, resolving at 2026-06-18 09:43 UTC; a fix was deployed and full recovery confirmed.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 64%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Increased latency and error rates
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-06-18 08:05 UTC
Before: not present
After: 2026-06-18 09:43 UTC
Before: not present
After: REST API, REST API - REST API
Before: not present
After: We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Before: not present
After: We've identified the issue, we're continuing working on rolling out a fix. Impact is restricted to the builds create API for a subset of tenants.
Before: not present
After: A fix has been deployed, services are recovering.
Before: not present
After: We have seen a full recovery of services.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Increased latency and error rates (this change) Buildkite reported a major degradation affecting the REST API’s builds create API for a subset of tenants, with increased latency and error rates. It started at 2026-06-18 08:05 UTC and lasted 1 h 37 min, resolving at 2026-06-18 09:43 UTC; a fix was deployed and full recovery confirmed. | Major incident |
All Buildkite changes → · Buildkite profile · Markdown · JSON