Companies / Astronomer / incidents
Astronomer: Tasks remain in the queued state on some worker queues.
Astronomer reported a major Astro degradation: tasks remained queued on some worker queues because autoscaling resource names exceeding Kubernetes’ 63-character limit were rejected, preventing worker scaling for a very small number of customers. It started at 2026-09-08 12:33 UTC and resolved at 2026-09-08 20:09 UTC, lasting 7 h 35 min; a fix was implemented.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Scheduling and Running DAGs and TasksScheduling and Running DAGs and Tasks
Timeline of updates
- identified
We have identified the cause. On Astro, each worker queue is backed by a Kubernetes autoscaling resource whose name is built from a fixed platform prefix, the deployment release name, and the worker queue name. For a very small number of customers, where that combined name exceeds the Kubernetes 63-character limit, the autoscaling resource is rejected at creation. As a result, workers do not scale for the affected queue and tasks routed there remain in the queued state. This is why the limit can be reached even when the worker queue name itself looks short: the release name and platform prefix already consume a large portion of the 63 characters before the queue name is appended. Workaround: If affected, until the permanent fix is deployed, please keep worker queue name at or below 10 characters. We are preparing a permanent fix. Further updates to follow.
- monitoring
A fix has been implemented, and we are monitoring the results.
- resolved
The incident has been resolved.
Rewindex changes for this incident
- Major incident: Tasks remain in the queued state on some worker queues. 2026-09-08 12:33 UTC
Impact, status and times are the vendor’s own, as last archived. View on status.astronomer.io (opens in new tab). Astronomer profile · Recent Astronomer incidents · Markdown · JSON