Dagster change history

Dagster reported a major Dagster Cloud UI outage affecting multiple users, starting 2025-02-27 10:52 UTC and resolved at 2025-02-27 11:29 UTC (37 min). A compute log download fault caused web-server timeouts; a fix restored access, while STDOUT/STDERR tabs remained temporarily disabled before later restoration. Agents, runs, and automations were unaffected.

Dagster reported a major API and Dagster Cloud UI degradation from 2025-02-07 15:37 UTC to 2025-02-07 16:11 UTC (34 min): some runs with concurrency keys lacking explicit configuration or a default limit were blocked, and concurrency settings had UI issues. Fixes restored progress, but previously blocked runs might still require manual termination.

Dagster reported a critical outage affecting its API and Dagster Cloud UI starting 2024-09-30 23:36 UTC, lasting 48 min and resolved at 2024-10-01 00:25 UTC. A storage-layer configuration error prevented service access; a fix was implemented, but some runs may have been delayed or failed.

Dagster reported major degradation of the Dagster Cloud UI run timeline due to overfetching runs, affecting timeline performance. The incident started at 2024-08-01 19:31 UTC and lasted 1 h 48 min, resolving at 2024-08-01 21:19 UTC; a fix was implemented at 08-01 21:08 UTC.

Dagster Cloud UI was unavailable in a critical-impact incident starting 2024-06-03 21:56 UTC and resolved at 2024-06-03 22:01 UTC, with a reported duration of 4 min. Dagster attributed the loss of UI access to a load balancer configuration change and rolled it back.

Some Dagster Plus Serverless runs failed to start with an "Agent detected that run worker failed." error during a major-impact incident from 2024-05-14 18:53 UTC to 2024-05-14 20:16 UTC, lasting 1 h 22 min. Dagster rolled out additional automatic retries; its cloud provider identified a causal change and began rolling it back. Dagster confirmed the fix was live.

A major incident affecting API and Dagster Cloud UI broke login and interactive web use for organizations whose firewalls allow-listed dagster.cloud, after erroneous redirects to dagster.plus. It ran from 2024-04-17 13:20 UTC to 2024-04-17 13:53 UTC (33 min); Dagster rolled back the redirects and implemented a fix, advising users with lingering login trouble to clear their browser cache.

Some Dagster Cloud API requests and a small number of runs failed with Internal Server Error exceptions due to a storage-engine timeout. The vendor rated the incident major; it ran from 2024-04-11 17:02 UTC to 2024-04-11 21:25 UTC, lasting 4 h 23 min. A fix was implemented and the incident resolved.

Dagster Cloud’s API incident was classified major, started at 2024-02-26 17:44 UTC, and was resolved at 2024-02-26 18:05 UTC (20 min). Dagster reported an outage from 17:28 UTC - 17:39 UTC while rotating production secrets; traffic had resumed and recovery was monitored.

Some Dagster Serverless runs failed to launch in a major outage starting 2023-09-18 18:07 UTC and resolved at 2023-09-19 04:03 UTC, lasting 9 h 55 min. Dagster reported AWS networking issues in two availability zones caused about 30% of new runs to fail container start; it deployed a fallback to stable zones, and the underlying issue appeared resolved.

Dagster reported a critical incident starting 2023-09-18 12:46 UTC, with a computed duration of 0 min. The vendor described a 90-second disruption during cloud-provider database maintenance as a primary database failed over to a secondary; Dagster operations during that window would have failed. The incident was resolved, with all systems back to normal.

Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved.

Dagster reported a major API incident preventing Serverless runs from launching, starting 2023-08-03 20:29 UTC and resolved at 2023-08-03 21:23 UTC (53 min). Runs were launching successfully as of 20:50 UTC; Dagster attributed the issue to AWS EC2 API errors and latency affecting AWS Fargate, used by Dagster Cloud Serverless.

Dagster Cloud had a vendor-rated critical partial outage affecting DNS routing, API and Dagster Cloud UI, starting 2023-06-14 17:44 UTC and lasting 16 min until resolution at 2023-06-14 18:00 UTC. DNS records were updated and propagating; users still unable to access Dagster Cloud were advised to clear their DNS cache.

Dagster reported a major API incident with jobs not starting, beginning 2023-05-03 22:58 UTC and resolved at 2023-05-03 23:29 UTC (30 min). A cluster node upgrade overlapped with the weekly release, temporarily disabling daemons controlling run queuing, alerting, schedules, and sensors; Dagster reverted a recent change and confirmed resolution.

Dagster Cloud reported a major API incident from 2023-01-27 19:14 UTC to 2023-01-27 20:50 UTC, lasting 1 h 36 min. A GraphQL-layer error caused sensors and runs using the k8s_job_executor to fail; a fix was implemented and the incident was resolved.

Dagster reported a critical API incident disrupting schedulers and sensors from 2022-11-02 19:04 UTC to 2022-11-02 20:34 UTC, lasting 1 h 30 min. Dagster reported an agent request queue backend upgrade at the start; a fix was implemented by 2022-11-02 20:29 UTC, and the incident was resolved.

Dagster reported a major Serverless agent outage that prevented some Serverless agents from launching runs. It started and was resolved at 2022-10-13 23:00 UTC, with a computed duration of 0 min; Dagster identified and rolled back the change responsible.

Dagster reported a major Cloud scheduler outage affecting the API component, starting at 2022-10-06 23:33 UTC and resolved at 2022-10-06 23:44 UTC (computed duration: 10 min). The scheduler returned to normal operation; no cause was stated.

Dagster Cloud reported elevated error rates with major impact. The incident was marked resolved, but its listed resolution (2022-08-19 15:39 UTC) precedes its start (2022-08-19 16:04 UTC); duration is unknown. The only update said Dagster was investigating, with no cause or fix stated.