# Astronomer: Major incident: Permissions silently dropped from custom roles

**Summary:** Astronomer reported a major outage from 2026-09-17 17:54 UTC to 2026-09-17 21:50 UTC, lasting 3 h 56 min. A regression from a change rolled out that day silently dropped custom-role permissions, causing reported deploy failures for tokens using those roles; re-adding deployment.deploys.create mitigated failures. All custom roles were restored, and the incident was resolved. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2026-09-17 17:54 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://status.astronomer.io/incidents/01M2R82A38HH3X91BKSJH8WR22](https://status.astronomer.io/incidents/01M2R82A38HH3X91BKSJH8WR22)
- [Company: Astronomer](https://rewindex.com/company/astronomer.md)
- Versions: before (none: first archived) → after incident\_version:135/01M2R82A38HH3X91BKSJH8WR22@23da210862e4
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Permissions silently dropped from custom roles
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2026-09-17 17:54 UTC
```

**5. Added: Resolved time**

```diff
+ 2026-09-17 21:50 UTC
```

**6. Added: Update** (identified) · 2026-09-17 17:54 UTC

```diff
+ We have identified a regression affecting custom roles due to a change rolled out today. This resulted in permissions being dropped from custom roles silently.  We have received reports of deploys linked to tokens using custom roles not working. Until the fix has been rolled out, deploy failures can be mitigated by re-adding the deployment.deploys.create permission to your custom role. You may also manually restore any other permissions that were silently dropped. We are working on fixing this regression for all affected custom roles.
```

**7. Added: Update** (monitoring) · 2026-09-17 19:29 UTC

```diff
+ All custom roles have been restored to their previous functionality and full operation has been restored.
```

**8. Added: Update** (resolved) · 2026-09-17 21:50 UTC

```diff
+ No further issues have been found, so this is being marked as resolved.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2026-09-17 17:54 UTC | Major incident: Permissions silently dropped from custom roles (this change) | outage |

All Astronomer changes: [timeline](https://rewindex.com/company/astronomer/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i31d62fbb771ac8 · Markdown: https://rewindex.com/change/i31d62fbb771ac8.md · JSON: https://rewindex.com/api/v1/changes/i31d62fbb771ac8 · Generated 2026-10-18T01:30:32.625Z