# Dagster: Critical incident: Scheduler and sensor disruptions

**Summary:** Dagster reported a critical API incident disrupting schedulers and sensors from 2022-11-02 19:04 UTC to 2022-11-02 20:34 UTC, lasting 1 h 30 min. Dagster reported an agent request queue backend upgrade at the start; a fix was implemented by 2022-11-02 20:29 UTC, and the incident was resolved. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2022-11-02 19:04 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/xbkkxwwj0092](https://stspg.io/xbkkxwwj0092)
- [Company: Dagster](https://rewindex.com/company/dagster.md)
- Versions: before (none: first archived) → after incident\_version:141/k51q9dr5rkzv@74164757c680
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Scheduler and sensor disruptions
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2022-11-02 19:04 UTC
```

**5. Added: Resolved time**

```diff
+ 2022-11-02 20:34 UTC
```

**6. Added: Affected components**

```diff
+ API
```

**7. Added: Update** (identified) · 2022-11-02 19:04 UTC

```diff
+ The issue has been identified and a fix is being implemented.
```

**8. Added: Update** (identified) · 2022-11-02 19:44 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**9. Added: Update** (monitoring) · 2022-11-02 20:29 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**10. Added: Update** (resolved) · 2022-11-02 20:34 UTC

```diff
+ We upgraded our agent request queue backend at 2022-11-02 19:04 UTC. During upgrades, Dagster Cloud's backend services read messages from the old queue and write messages onto the new queue. Once the old queue is empty, Dagster Cloud's services begin reading messages only from the new queue. When rolling the new queue configuration out to Dagster Cloud's backend services, our scheduling services did not recognize the new configuration. These services continued to look only at the old queue. This means new agent requests were being written to the new queue, but our scheduler services - which control things like executing schedules, sensors, and launching runs - continued to only read from the old queue. Customers may need to manually retry runs that launched but were marked as failed during the outage window because Dagster Cloud never acknowledged the launch event. Schedules and sensors will resume automatically. To mitigate this in the future, when performing similar operations, we plan to use immutable configurations that will preclude services from continuing to run with an old configuration.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2022-11-02 19:04 UTC | Critical incident: Scheduler and sensor disruptions (this change) | outage |

All Dagster changes: [timeline](https://rewindex.com/company/dagster/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i9272363d3fd693 · Markdown: https://rewindex.com/change/i9272363d3fd693.md · JSON: https://rewindex.com/api/v1/changes/i9272363d3fd693 · Generated 2026-10-18T03:55:49.453Z