Changes / Dagster / Reports of jobs not starting
Dagster: Major incident: Reports of jobs not starting
Summary
Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Reports of jobs not starting
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2023-05-03 22:58 UTC
Before: not present
After: 2023-05-03 23:29 UTC
Before: not present
After: API
Before: not present
After: We have reverted a recent change and believe this should resolve the issue. We will provide an update after further investigation.
Before: not present
After: The issue is confirmed as resolved, though we are continuing to investigate the issue.
Before: not present
After: This incident has been resolved.
An operation to upgrade the nodes of our cluster overlapped with our weekly release. Interaction between the two operations caused the Dagster daemons that control run queuing, alerting, schedules, and sensors to be temporarily unavailable.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Reports of jobs not starting (this change) Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution. | Major incident |