Dagster: dagster.plus redirect breaks login for some organizations

Impact: majorStatus: resolvedStarted 2024-04-17 13:20 UTCResolved 2024-04-17 13:53 UTC (33 min)3 updatesSource: stspg.io

Summary

A major incident affecting API and Dagster Cloud UI broke login and interactive web use for organizations whose firewalls allow-listed dagster.cloud, after erroneous redirects to dagster.plus. It ran from 2024-04-17 13:20 UTC to 2024-04-17 13:53 UTC (33 min); Dagster rolled back the redirects and implemented a fix, advising users with lingering login trouble to clear their browser cache.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: APIDagster Cloud UI

Timeline of updates

identified2024-04-17 13:20 UTC
In preparation for our Dagster+ launch, we erroneously began redirecting dagster.cloud to dagster.plus. We are rolling back this change now. If you continue to have trouble logging in, you may need to clear your browser's cache.
monitoring2024-04-17 13:53 UTC
A fix has been implemented and we are monitoring the results.
resolved2024-04-17 13:53 UTC
In anticipation of today's Dagster+ launch event, we added a second domain to our app: dagster.plus. Last night, we erroneously started redirecting certain requests from dagster.cloud to dagster.plus. As a result, organizations who explicitly allow-list dagster.cloud in their corporate firewalls saw redirected requests to dagster.plus fail. Affected requests included: - Interactive use of the Dagster web interface - GraphQL queries authenticated with a user token (for example, many CI/CD jobs) Agent interactions (including launching runs and storing events) were not redirected and continued to operate as expected. Organizations without these firewall settings were not affected. Once detected, we began working on a way to selectively disable redirects for affected organizations. Shortly after, we removed redirects entirely for all organizations. However, many browsers permanently cache redirects. If you continue to have trouble accessing dagster.cloud please clear your browser cache and try again. Internally, we're conducting a full postmortem and deciding how to more safely introduce our new domain name in the future.

Rewindex changes for this incident

Major incident: dagster.plus redirect breaks login for some organizations 2024-04-17 13:20 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Dagster profile · Recent Dagster incidents · Markdown · JSON