Before: not present
After: 1. Summary
From October 28, 2026, 10:14 AM PT to October 29, 2026, 9:12 PM PT, CI pipelines using VM-based runners intermittently hung or expired after initialization, often with no step logs or console output. The issue affected two Prod1 customers, with additional VM-runner customers identified for proactive follow-up across Prod1, Prod2, and Prod3. The trigger was message-queue backpressure caused by broker slowness following an infrastructure image upgrade in that layer; we mitigated by rolling back that change, which resolved the platform-side trigger within about two hours. Follow-up found the two affected customers were running older versions of our VM runner and its companion log-streaming component that predated a fix for a related deadlock issue, so the customer-facing resolution was to pin to a specific, validated runner version instead of relying on a floating "latest" tag.
2. Impact
CI jobs on VM runners could hang mid-run, time out after long waits , or expire without usable logs.
This blocked deployments for affected pipelines and slowed diagnosis, since failures were often silent
Two Prod1 customers were confirmed affected; a broader set of VM-runner customers across Prod1, Prod2, and Prod3 were identified as potentially exposed and proactively contacted.
Impact was limited to CI workflows using VM-based runners; other execution types were not affected.
3. Root Cause
Our CI VM-runner architecture streams step logs through a live-logging pipeline backed by a message-queue layer. During this incident, that queue experienced backpressure, following an infrastructure image upgrade on one of the queue's brokers. This degraded the live-log streaming path platform-wide, causing some in-flight log writes to stall.
However, this platform-level degradation was transient and self-healed once we rolled back the image upgrade. It only turned into multi-hour, customer-visible hangs for customers who were running older builds of our VM runner and its companion log-streaming helper component,specifically, builds that predated a fix for a known deadlock condition in how that component handled interrupted log streams. Under normal queue conditions, this defect was dormant; the queue backpressure was enough to trigger it on those older builds, causing their executions to hang on an internal request until a roughly 4-hour timeout expired.
Both confirmed affected customers were using runner versions older than the version containing the fix, and in one case the customer believed they were on the latest available image but were, in practice, still running an older pulled version — because our "latest" tag at the time did not reliably correspond to the newest, QA-validated build.
4. Resolution
Identified that failures were specific to VM-runner executions hanging after initialization, and separated this from an unrelated initialization-stage incident from the prior day.
Traced message-queue producer errors and broker slowness to a recent infrastructure image upgrade on that queue layer.
Rolled back that image upgrade, which brought queue/log-streaming error rates to zero within about two hours of incident start.
Confirmed, with the two affected customers, the specific runner and log-streaming component versions they were running.
Recommended and confirmed a specific, pinned runner version containing the deadlock fix as the customer-facing resolution.
Identified additional VM-runner customers for proactive outreach and validation across Prod1, Prod2, and Prod3.
5. Preventive Actions
Surface the VM runner version directly in pipeline execution logs and metadata, so this no longer has to be manually confirmed with customers during an incident.
Publish guidance recommending customers pin to an explicit runner version rather than relying on "latest".
Add detection for post-initialization stalls, specifically keyed to the ~4-hour timeout signature seen in this incident.
Build a queryable inventory of VM-runner customers and the versions they're running, to support faster, more targeted response in future incidents and proactive upgrade outreach.