Struck in red is what Dagster removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: Root cause analysis
On October 7 and 8, 2026, some Dagster+ customers experienced failed run launches, timeouts, and delays in recording run events. During a release on October 7, the Redis service that carries requests between Dagster+ and customer agents became overloaded. Our investigation also identified inefficient reuse of database connections in the service that records run events, which caused recurring delays over the following days.
We’re sorry for the disruption and for the delay in communicating with customers during the incident.
Customer Impact
Beginning at approximately 19:15 UTC on October 7, some customers experienced requests timing out or runs failing to start. The initial disruption subsided around 19:39 UTC, but intermittent delays continued over the following days.
During this period, Dagster+ sometimes took longer to record updates from running jobs. Customers also reported missed runs after the initial disruption. These interruptions were particularly consequential for workflows that needed to run at precise intervals or collect information that could not be retrieved later.
We applied several patches over the following days to reduce the impact, then resolved the recurring delays in recording run events with a fix on October 12. We confirmed the improvement through peak traffic on October 13, with no further latency warnings during that verification period.
What happened
October 7: Run launch failures
During the Dagster+ 1.13.26 rollout, the Redis service used for request storage reached its capacity. The service had limited headroom before the rollout, and the number of application instances restarting and reconnecting during the release increased load.
As Redis became saturated, requests slowed or timed out. Application instances waiting on Redis consumed very little CPU; our automatic scaling interpreted that to be spare capacity and reduced the number of application instances, concentrating more work on the remaining instances.
Redis recovered as its buffers cleared. Customer reports show that missed runs continued after request-storage Redis recovered. Engineering also observed elevated event-write latency during this period, although the available evidence does not establish a direct link to every reported missed run.
October 8: Intermittent slowness
The service that records run events repeatedly opened and closed database connections under load. Repeated recreation of Postgres connections was expensive and drove the latency spikes. Under high load, the service repeatedly created extra database connections and discarded them after use, spending time opening connections instead of processing run events. We made several more changes to reduce the impact of load on the cluster, and changed how the service reused database connections on October 12. This resolved the recurring delays, and we confirmed the improvement through peak traffic on October 13.
Prevention and follow-up
We increased the capacity of the request-storage Redis service, upgraded it to Valkey 8, increased the CPU limit on the event-writing service, and adjusted that service’s scaling behavior to reduce how quickly instances are removed. We are also evaluating limits to prevent Redis connection pools from growing without bounds, and will roll it out after testing in pre-production.
Our planned follow-up work includes reviewing recurring capacity warnings, strengthening capacity checks during releases, and improving alerts for slow requests and growing backlogs. We also plan to connect customer support reports more closely with incident investigations and coordinate updates to affected customers.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.
Incident updateDegraded agent communication delayed schedules and automationsthis change
Cite this
Incident updated: Degraded agent communication delayed schedules and automations on Dagster’s status page, changed 2026-10-29 18:33 UTC (change 1 of 1).
Title
Incident updated: Degraded agent communication delayed schedules and automations