Companies/Dagster/Incidents

Dagster: Cloud application cluster outage

Affected components: APIDagster Cloud UI

Timeline of updates

1. investigating2026-10-22 17:14 UTC

   We are experiencing an issue with our production application cluster, impacting all Dagster+ cloud customers. We are currently investigating and working on a resolution.

2. investigating2026-10-22 17:39 UTC

   We've identified the core issue, and are actively working to remediate. We are rolling back our deploy, and increasing the size of our database cluster to manage the increased load.

3. investigating2026-10-22 17:50 UTC

   The rollback and upgraded database have completed and the system is operational.

4. resolved2026-10-22 17:51 UTC

   This incident has been resolved.

5. postmortem2026-10-30 13:44 UTC

   Dagster+ UI timeouts and run failures
   On October 22, 2026, Dagster+ customers in our US region experienced UI and API timeouts, failed runs, and delayed schedules and sensors for about two hours. The cause was a change in that day's release that made a frequently run database query much slower, which overloaded our primary database.
   We're sorry for the disruption, and especially for its effect on pipelines that depend on running on time.
   Customer Impact
   From approximately 16:40 UTC, customers saw slow or failing pages in the Dagster+ UI, 504 errors and timeouts from the API, and failed runs. At the peak, roughly half as many runs completed successfully as normal. Schedules, sensors, and automation evaluations were delayed or skipped during the same period.
   Most symptoms subsided after we rolled back the release at about 17:40 UTC. Some customers continued to see intermittent errors until about 18:45 UTC, for the reason described below.
   What happened
   The October 22 release included a change to a query that runs every time a step in a run starts, which makes it one of the most frequently run queries in Dagster+. The change was logically correct, but the production database chose a different, much less efficient query plan to execute it than the pre-production environments it was tested against, so each run of it took about 50 times longer than before.
   The extra load pushed the database to its CPU limit within a few minutes of the release. Requests across Dagster+ then slowed or timed out, including those that start runs and record their progress. Because runs whose steps couldn't start retried, the load on the database increased further.
   Recovery
   We rolled the release back at about 17:35 UTC, which removed the slow query. At the same time, we moved the database to a server with twice the compute capacity.
   While the database was slow, our automatic scaling had increased the number of web server instances to its maximum, because the instances looked busy waiting for the database. When the database came back on its new server, all of those instances reconnected at once and exceeded the database's limit on concurrent connections. That caused the intermittent errors that continued until about 18:45 UTC, when we manually lowered the scaling limit and connections returned to normal.
   Prevention and follow-up
   We've completed the following:
   Reverted the query change.
   Doubled the compute capacity of the primary database.
   Lowered the maximum number of web server instances to stay within the database's connection limit.
   Routed web server database connections through a connection pooler, so that scaling up web servers can't exhaust the database's connections.
   We're also working on:
   Redistributing database workloads off of this primary instance
   Requiring that changes to database queries be tested against production-scale data before release, with automated review that flags query changes lacking that evidence.
   Adding alerts on database CPU and connection usage, and checks during each release that pause it if database performance regresses.
   Improving our process for updating customers during an incident.

Changes Rewindex caught on this incident

• Oct 22Critical incident: Cloud application cluster outage
• Oct 22Incident updated: Cloud application cluster outage
• Oct 22Incident updated: Cloud application cluster outage
• Oct 30Incident updated: Cloud application cluster outage

Impact, status and times are the vendor’s own, as last archived. View on dagstercloud.statuspage.io (opens in new tab).

Dagster profileRecent Dagster incidentsAll outages on October 22, 2026Dagster in October 2026MarkdownJSON