Changes / Buildkite / Delayed notifications
Buildkite: Major incident: Delayed notifications
Summary
Buildkite reported a major incident delaying build and job notifications for a subset of customers, affecting email, GitHub commit status, Slack and webhooks. It started at 2026-05-28 20:20 UTC and was resolved at 2026-05-28 21:18 UTC; the computed duration was 57 min. Delayed notifications also delayed downstream CI, merge or deployment workflows for some customers.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Delayed notifications
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-05-28 20:20 UTC
Before: not present
After: 2026-05-28 21:18 UTC
Before: not present
After: Email Notifications, GitHub Commit Status Notifications, Notifications - Email Notifications, Notifications - GitHub Commit Status Notifications, Notifications - Slack Notifications, Notifications - Webhook Notifications, Slack Notifications, Webhook Notifications
Before: not present
After: We are investigating delays to build and job notifications for a subset of customers.
Before: not present
After: We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Before: not present
After: This incident has been resolved.
Before: not present
After: ## Service Impact
Customers experienced delayed Buildkite notification delivery. The customer impact varied depending on how those notifications are used. For some customers, delayed notifications also delayed downstream CI, merge, or deployment workflows.
## Incident Summary
On 28 May, Buildkite experienced elevated notification delivery latency after part of our notification-processing infrastructure became underprovisioned. This happened because the Prometheus service used by our EKS autoscaling path ran out of storage, which meant some EKS-based workers could not autoscale correctly while queues were growing. We mitigated the incident by moving affected workloads back to our previous ECS-based infrastructure and manually increasing worker capacity. Recovery took longer than expected because the rollback path did not fully handle this scenario.
### Impact window 1
At 20:01 UTC, notification-processing workers became underprovisioned and notification delivery latency increased. We detected the issue through internal queue latency monitoring and began shifting affected workloads from EKS back to ECS. This rollback took longer than expected because the ECS services we were rolling back to were not ready to immediately take the full load. Engineers had to manually adjust scaling configuration and worker counts while the incident was active. Notification latency recovered for most customers by 21:00 UTC.
### Impact window 2
A second, shorter impact window occurred between 22:12 UTC and 22:40 UTC for a subset of customers. After the first recovery, some workloads were still running on EKS and had started autoscaling again after Prometheus recovered. We incorrectly believed those workloads were no longer serving traffic. When we reconciled our infrastructure configuration, those EKS workloads were scaled down before their ECS equivalents had been fully scaled up. This caused another period of underprovisioning for some notification-processing workers. We resolved it by completing the rollback and scaling the remaining affected ECS services.
### Customer Impact
The impact was not identical for every customer. For customers who use Buildkite notifications as an input to other CI or deployment systems, notification latency can delay those downstream workflows. Some customers also experienced secondary or longer-running effects based on the specific notification types, retry behaviour, or integrations involved. We are following up directly with affected customers where their impact differed from the general incident.
## Changes we're making
We have made the following immediate changes:
* Increased Prometheus storage capacity and reconciled that change in infrastructure-as-code.
* Added monitoring to alert before Prometheus storage exhaustion can affect autoscaling.
* Moved affected notification-processing workloads back to known-good ECS capacity.
* Fixed GitHub notification retry behaviour for a class of errors that could cause repeated retries and extend notification delays.
We are also making the following reliability improvements:
* Hardening the EKS-to-ECS rollback process so it verifies destination capacity, autoscaling configuration, and traffic movement before and during rollback.
* Reviewing other EKS control-plane dependencies, including KEDA and Karpenter, to ensure their CPU, memory, and storage allocations are appropriate for production load.
* Reassessing the order and pace of future EKS migrations so customer-critical workloads move more gradually and with clearer settling periods.
* Improving customer-level monitoring for notification delivery latency, so we can detect customer-impacting regressions earlier.
* Reviewing which notification types are on the scheduling or CI hot path for customers, and whether they need tighter latency expectations, separate queueing, or more specific alerting than general notification work.
## Areas we are improving: incident communication
During this incident, our public status page did not reflect customer-visible impact as quickly or clearly as it should have. In particular, notification delivery latency can affect customers differently depending on how notifications are used in their CI and deployment workflows.
We are improving how we communicate during notification latency incidents by:
* Updating the status page earlier when notification latency is likely to affect customer workflows
* Making status page updates clearer about the customer-visible impact, not just the affected internal service
* Improving internal escalation paths for customers who report critical CI impact before the incident is fully understood
* Using customer-level notification latency monitoring to help identify affected customers sooner
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delayed notifications (this change) Buildkite reported a major incident delaying build and job notifications for a subset of customers, affecting email, GitHub commit status, Slack and webhooks. It started at 2026-05-28 20:20 UTC and was resolved at 2026-05-28 21:18 UTC; the computed duration was 57 min. Delayed notifications also delayed downstream CI, merge or deployment workflows for some customers. | Major incident |
All Buildkite changes → · Buildkite profile · Markdown · JSON