Struck in red is what Dagster removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Dagster+ UI timeouts and run failures
On October 22, 2026, Dagster+ customers in our US region experienced UI and API timeouts, failed runs, and delayed schedules and sensors for about two hours. The cause was a change in that day's release that made a frequently run database query much slower, which overloaded our primary database.
We're sorry for the disruption, and especially for its effect on pipelines that depend on running on time.
Customer Impact
From approximately 16:40 UTC, customers saw slow or failing pages in the Dagster+ UI, 504 errors and timeouts from the API, and failed runs. At the peak, roughly half as many runs completed successfully as normal. Schedules, sensors, and automation evaluations were delayed or skipped during the same period.
Most symptoms subsided after we rolled back the release at about 17:40 UTC. Some customers continued to see intermittent errors until about 18:45 UTC, for the reason described below.
What happened
The October 22 release included a change to a query that runs every time a step in a run starts, which makes it one of the most frequently run queries in Dagster+. The change was logically correct, but the production database chose a different, much less efficient query plan to execute it than the pre-production environments it was tested against, so each run of it took about 50 times longer than before.
The extra load pushed the database to its CPU limit within a few minutes of the release. Requests across Dagster+ then slowed or timed out, including those that start runs and record their progress. Because runs whose steps couldn't start retried, the load on the database increased further.
Recovery
We rolled the release back at about 17:35 UTC, which removed the slow query. At the same time, we moved the database to a server with twice the compute capacity.
While the database was slow, our automatic scaling had increased the number of web server instances to its maximum, because the instances looked busy waiting for the database. When the database came back on its new server, all of those instances reconnected at once and exceeded the database's limit on concurrent connections. That caused the intermittent errors that continued until about 18:45 UTC, when we manually lowered the scaling limit and connections returned to normal.
Prevention and follow-up
We've completed the following:
Reverted the query change.
Doubled the compute capacity of the primary database.
Lowered the maximum number of web server instances to stay within the database's connection limit.
Routed web server database connections through a connection pooler, so that scaling up web servers can't exhaust the database's connections.
We're also working on:
Redistributing database workloads off of this primary instance
Requiring that changes to database queries be tested against production-scale data before release, with automated review that flags query changes lacking that evidence.
Adding alerts on database CPU and connection usage, and checks during each release that pause it if database performance regresses.
Improving our process for updating customers during an incident.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.
Dagster reported a critical production application cluster outage affecting API and Dagster Cloud UI for all Dagster+ cloud customers, starting 2026-10-22 17:14 UTC. It was still open when archived, open for 0 min at the last update; Dagster was investigating and working on a resolution.
Cite this
Incident updated: Cloud application cluster outage on Dagster’s status page, changed 2026-10-30 13:53 UTC (change 4 of 4).