Changes / v0 / Delays in Drains, Observability, and Workflow in Washington (iad1)
v0: Major incident: Delays in Drains, Observability, and Workflow in Washington (iad1)
Summary
v0 reported a major degradation in Washington (iad1): some customers experienced delayed Drains delivery, logs, metrics and traces, with Workflow runs remaining pending. It started at 2026-10-10 00:15 UTC and resolved at 2026-10-10 01:37 UTC, lasting 1 h 21 min. A fix restored delivery, delayed data was processed, and all services recovered.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
What changed
Before: not present
After: Delays in Drains, Observability, and Workflow in Washington (iad1)
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-10-10 00:15 UTC
Before: not present
After: 2026-10-10 01:37 UTC
Before: not present
After: Drains, Logs, Observability, Observability - Drains, Observability - Logs, Observability - Observability, Queues, Workflow
Before: not present
After: We've identified an issue where some customers may experience delayed drain delivery, delayed logs, metrics, and traces, and Workflow runs that remain pending in Washington (iad1).
We are investigating this issue. We will provide additional updates as they become available.
Before: not present
After: A fix has been implemented and delivery has resumed. Delayed Drains deliveries and Observability data are being processed. We are continuing to monitor.
Before: not present
After: Delayed Drains deliveries, Observability data, and Build logs have been processed now. We are continuing to monitor and we will provide additional updates as they become available.
Before: not present
After: All services have recovered.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delays in Drains, Observability, and Workflow in Washington (iad1) (this change) v0 reported a major degradation in Washington (iad1): some customers experienced delayed Drains delivery, logs, metrics and traces, with Workflow runs remaining pending. It started at 2026-10-10 00:15 UTC and resolved at 2026-10-10 01:37 UTC, lasting 1 h 21 min. A fix restored delivery, delayed data was processed, and all services recovered. | Major incident |