Changes / Dagster / Scheduler and sensor disruptions
Dagster: Critical incident: Scheduler and sensor disruptions
Summary
Dagster reported a critical API incident disrupting schedulers and sensors from 2022-11-02 19:04 UTC to 2022-11-02 20:34 UTC, lasting 1 h 30 min. Dagster reported an agent request queue backend upgrade at the start; a fix was implemented by 2022-11-02 20:29 UTC, and the incident was resolved.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Scheduler and sensor disruptions
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2022-11-02 19:04 UTC
Before: not present
After: 2022-11-02 20:34 UTC
Before: not present
After: API
Before: not present
After: The issue has been identified and a fix is being implemented.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: We upgraded our agent request queue backend at 2022-11-02 19:04 UTC.
During upgrades, Dagster Cloud's backend services read messages from the old queue and write messages onto the new queue. Once the old queue is empty, Dagster Cloud's services begin reading messages only from the new queue.
When rolling the new queue configuration out to Dagster Cloud's backend services, our scheduling services did not recognize the new configuration. These services continued to look only at the old queue.
This means new agent requests were being written to the new queue, but our scheduler services - which control things like executing schedules, sensors, and launching runs - continued to only read from the old queue.
Customers may need to manually retry runs that launched but were marked as failed during the outage window because Dagster Cloud never acknowledged the launch event.
Schedules and sensors will resume automatically.
To mitigate this in the future, when performing similar operations, we plan to use immutable configurations that will preclude services from continuing to run with an old configuration.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Scheduler and sensor disruptions (this change) Dagster reported a critical API incident disrupting schedulers and sensors from 2022-11-02 19:04 UTC to 2022-11-02 20:34 UTC, lasting 1 h 30 min. Dagster reported an agent request queue backend upgrade at the start; a fix was implemented by 2022-11-02 20:29 UTC, and the incident was resolved. | Major incident |