Companies / Dagster / incidents

Dagster: Reports of jobs not starting

Impact: majorStatus: resolvedStarted Resolved (30 min)3 updatesSource: stspg.io (opens in new tab)
Summary

Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: API

Timeline of updates

  1. investigating

    We have reverted a recent change and believe this should resolve the issue. We will provide an update after further investigation.

  2. investigating

    The issue is confirmed as resolved, though we are continuing to investigate the issue.

  3. resolved

    This incident has been resolved. An operation to upgrade the nodes of our cluster overlapped with our weekly release. Interaction between the two operations caused the Dagster daemons that control run queuing, alerting, schedules, and sensors to be temporarily unavailable.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Dagster profile · Recent Dagster incidents · Markdown · JSON