Changes / Dagster / Dagster UI down for multiple users
Dagster: Major incident: Dagster UI down for multiple users
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Dagster UI down for multiple users
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-02-27 10:52 UTC
Before: not present
After: 2025-02-27 11:29 UTC
Before: not present
After: Dagster Cloud UI
Before: not present
After: We've received reports from multiple users having trouble accessing the Dagster Cloud UI. Our engineering team is currently investigating.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We have rolled out a fix and are monitoring service restoration. The Dagster+ UI is available again. STDERR and STDOUT tabs on runs are temporarily disabled.
Before: not present
After: We have rolled out a fix and are monitoring service restoration. The Dagster+ UI is available again. STDERR and STDOUT tabs on runs are temporarily disabled.
Before: not present
After: The Dagster+ UI is restored. Agents, runs, and automations are not affected. We will continue to work to restore STDOUT and STDERR in the UI during US business hours. In the meantime, compute logs will continue to upload but will not be accessible via the UI.
[UPDATE 1:10 pm EST] STDERR and STDOUT are available again in the UI for all users.
Before: not present
After: In addition to [structured event logs](https://docs.dagster.io/guides/monitor/logging/#structured-event-logs) that appear in the Dagster UI, Dagster supports [raw compute logs](https://docs.dagster.io/guides/monitor/logging/#raw-compute-logs) to capture STDERR and STDOUT.
During the affected window, a fault in our compute log download process resulted in timeouts and diminished responsiveness of our web servers.
We mitigated the issue by temporarily disabling compute log downloads before adjusting compute log behavior so that the conditions that led to the failure are no longer possible. Log functionality has since been fully restored. We've also adjusted our incident process to ensure status pages are posted more quickly in the future.
Please reach out if you have additional questions. Thank you for your patience.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Dagster UI down for multiple users (this change) | Major incident |