Search

Companies / Astronomer / incidents

Astronomer: KPO podspec changes for GPU-enabled nodes may cause KPO pod startup failures

Impact: minorStatus: resolvedStarted Resolved (1.5 h)3 updatesSource: status.astronomer.io (opens in new tab)

Affected components: Scheduling and Running DAGs and Tasks

Timeline of updates

  1. identified

    We've identified a change that enabled GPU support for KubernetesPodOperator pods had an unintended side effect when reading pod specs without proper limits set in the expected format. We're working on a fix, if you notice any unexpected failures for KPO tasks in your Astro deployments, please open a support ticket.

  2. monitoring

    We have rolled back the changes that triggered this issue and are monitoring the issue for confirmation that this issue is resolved.

  3. resolved

    Mitigations have been successful. A Platform release that corrects this errant behavior is being worked on now. Issue is resolved.

Impact, status and times are the vendor’s own, as last archived. View on status.astronomer.io (opens in new tab). Astronomer profile · Recent Astronomer incidents · Markdown · JSON