Rewindex

CompaniesRed Hat (incl. Quay.io)Incidents

Red Hat (incl. Quay.io): Quay.io HTTP 502 on Pull/Push

CriticalStatus: postmortemStarted Resolved (3.1 h)7 updatesstspg.io (opens in new tab)

Affected components: Quay.io - RegistryRegistryAPIQuay.io - API

Timeline of updates

  1. investigating

    We are currently investigating this issue.

  2. investigating

    Pulls have been restored. Pushes are unavailable while we continue to investigate.

  3. identified

    We have identified the issue as caused by a recent deployment. Our team has reverted the deployment, and are gradually rolling back traffic to our primary region to re-enable pushes while monitoring.

  4. monitoring

    A fix has been implemented and we are monitoring the results.

  5. monitoring

    We are continuing to monitor for any further issues.

  6. resolved

    This incident has been resolved.

  7. postmortem

    On May 7, a production deployment introduced a database locking conflict that caused Quay.io to become unhealthy. Image pulls were intermittently degraded for approximately 30 minutes and image pushes were unavailable for approximately 3 hours. The team failed over to a secondary region to restore pulls, then reverted the deployment to fully restore service. We are implementing multi-phase deployment rollouts, stricter canary processes, and health probe improvements to prevent recurrence.

Changes Rewindex caught on this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Red Hat (incl. Quay.io) profileRecent Red Hat (incl. Quay.io) incidentsMarkdownJSON