Changes / Dagster / Dagster Cloud Service
Dagster: Critical incident: Dagster Cloud Service
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Dagster Cloud Service
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2024-09-30 23:36 UTC
Before: not present
After: 2024-10-01 00:25 UTC
Before: not present
After: API, Dagster Cloud UI
Before: not present
After: We are currently experiencing an issue with Dagster Cloud's storage engine that is preventing access to the service. We are immediately investigating the issue and will post continued updates here.
Before: not present
After: We are in the process of rolling out a change that is intended to restore access. We will continue to post updates here as we work to restore service.
Before: not present
After: We are in the process of rolling out a change that is intended to restore access. We will continue to post updates here as we work to restore service.
Before: not present
After: The fix has been implemented and we are monitoring to confirm that service has been restored.
Before: not present
After: This incident has been resolved. Some runs may have been delayed or failed during the outage. We do not expect any further interruption of service.
Before: not present
After: During this weekend's maintenance window, we conducted a major update of our storage layer to enable continued scale and resiliency. The major update was completed over the weekend without a problem.
Yesterday, the team worked through a set of follow-up tasks to finalize the update. During this process, a configuration error was applied that led to a temporary outage in the new storage layer. The error was identified immediately, and steps were taken to restore service. The total outage was approximately 30 minutes.
During the outage, the Dagster\+ web interface was not accessible. Run instigators such as schedules and sensors were not run, and existing runs failed. When service was restored, run instigators were automatically restored and typically caught up automatically. Runs with retry policies were also evaluated and executed.
Going forward, we are taking a number of steps to improve and prevent future outages:
* The storage update that preceded this event, now complete, will enable higher availability of the storage layer.
* Additional steps are being put in place to make storage failover faster, and we are implementing changes to the service to allow existing runs to be resilient to short disruptions.
* Additional guardrails are being put in place to prevent configuration errors from being applied.
We appreciate that downtime has a significant impact for our customers. Please reach out to your Dagster Labs account team if you have further questions or concerns.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Dagster Cloud Service (this change) | Major incident |