Changes / Astronomer / Permissions silently dropped from custom roles

Astronomer: Major incident: Permissions silently dropped from custom roles

Summary

Astronomer reported a major outage from 2026-09-17 17:54 UTC to 2026-09-17 21:50 UTC, lasting 3 h 56 min. A regression from a change rolled out that day silently dropped custom-role permissions, causing reported deploy failures for tokens using those roles; re-adding deployment.deploys.create mitigated failures. All custom roles were restored, and the incident was resolved.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 76%)Changed Detected Source: status.astronomer.io

What changed

AddedTitle#1
Before: not present
After: Permissions silently dropped from custom roles
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2026-09-17 17:54 UTC
AddedResolved time#5
Before: not present
After: 2026-09-17 21:50 UTC
AddedUpdateidentified#6
Before: not present
After: We have identified a regression affecting custom roles due to a change rolled out today. This resulted in permissions being dropped from custom roles silently. We have received reports of deploys linked to tokens using custom roles not working. Until the fix has been rolled out, deploy failures can be mitigated by re-adding the deployment.deploys.create permission to your custom role. You may also manually restore any other permissions that were silently dropped. We are working on fixing this regression for all affected custom roles.
AddedUpdatemonitoring#7
Before: not present
After: All custom roles have been restored to their previous functionality and full operation has been restored.
AddedUpdateresolved#8
Before: not present
After: No further issues have been found, so this is being marked as resolved.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Major incident: Permissions silently dropped from custom roles (this change)
Astronomer reported a major outage from 2026-09-17 17:54 UTC to 2026-09-17 21:50 UTC, lasting 3 h 56 min. A regression from a change rolled out that day silently dropped custom-role permissions, causing reported deploy failures for tokens using those roles; re-adding deployment.deploys.create mitigated failures. All custom roles were restored, and the incident was resolved.
Major incident

All Astronomer changes → · Astronomer profile · Markdown · JSON