Companies / Dagster / incidents
Dagster: Some Dagster Plus Serverless Runs Failing to Start
Some Dagster Plus Serverless runs failed to start with an "Agent detected that run worker failed." error during a major-impact incident from 2024-05-14 18:53 UTC to 2024-05-14 20:16 UTC, lasting 1 h 22 min. Dagster rolled out additional automatic retries; its cloud provider identified a causal change and began rolling it back. Dagster confirmed the fix was live.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: API
Timeline of updates
- investigating
Some Dagster Plus Serverless runs are failing to start with an "Agent detected that run worker failed." error. We are working with our cloud provider to investigate the root cause of the problem as well as adding automatic retries for the underlying error that is causing the runs to fail to start.
- investigating
We are continuing to investigate the root cause of this issue with our cloud provider. We are rolling out a change to add additional automatic retries to runs that are affected by this issue.
- identified
Our cloud provider has identified the root cause of the problem and is in the process of rolling back the change that caused it. We'll continue to provide updates as we work to resolve the issue.
- resolved
The fix for this problem is now live.
Rewindex changes for this incident
- Major incident: Some Dagster Plus Serverless Runs Failing to Start 2024-05-14 18:53 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Dagster profile · Recent Dagster incidents · Markdown · JSON