Changes / v0 / Increase in Workflow runs stuck as pending
v0: Major incident: Increase in Workflow runs stuck as pending
Summary
A major Workflow degradation left some recently deployed projects stuck pending and not delivering queued messages from 2026-06-25 07:45 UTC to 2026-06-25 13:43 UTC, lasting 5 h 58 min. A fix was rolled out; the interim workaround was an instant rollback to a working deployment created before Jun 24 21:00 UTC to resume new runs.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 52%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Increase in Workflow runs stuck as pending
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-06-25 07:45 UTC
Before: not present
After: 2026-06-25 13:43 UTC
Before: not present
After: Workflow
Before: not present
After: We are currently investigating an issue where some recently deployed Workflow projects are stuck as pending and not delivering queued messages as expected.
Before: not present
After: The issue has been identified and a fix is being implemented.
Before: not present
After: The issue has been identified. We are continuing to work on a fix for this issue.
Before: not present
After: We are still working on a fix for this issue. While we fix the issue, users experiencing pending Workflow runs can trigger an instant rollback to a working deployment created before Jun 24 21:00 UTC to resume operation on new runs.
Before: not present
After: We are currently rolling a fix for the issue. We are monitoring the results.
Before: not present
After: The fix has been fully rolled out and we are monitoring the results.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Increase in Workflow runs stuck as pending (this change) A major Workflow degradation left some recently deployed projects stuck pending and not delivering queued messages from 2026-06-25 07:45 UTC to 2026-06-25 13:43 UTC, lasting 5 h 58 min. A fix was rolled out; the interim workaround was an instant rollback to a working deployment created before Jun 24 21:00 UTC to resume new runs. | Major incident |