After: Pulls have been restored. Pushes are unavailable while we continue to investigate.
AddedUpdateidentified#9
Before: not present
After: We have identified the issue as caused by a recent deployment. Our team has reverted the deployment, and are gradually rolling back traffic to our primary region to re-enable pushes while monitoring.
AddedUpdatemonitoring#10
Before: not present
After: A fix has been implemented and we are monitoring the results.
AddedUpdatemonitoring#11
Before: not present
After: We are continuing to monitor for any further issues.
AddedUpdateresolved#12
Before: not present
After: This incident has been resolved.
AddedUpdatepostmortem#13
Before: not present
After: On May 7, a production deployment introduced a database locking conflict that caused Quay.io to become unhealthy. Image pulls were intermittently degraded for approximately 30 minutes and image pushes were unavailable for approximately 3 hours. The team failed over to a secondary region to restore pulls, then reverted the deployment to fully restore service. We are implementing multi-phase deployment rollouts, stricter canary processes, and health probe improvements to prevent recurrence.
Rewindex commits each version of an incident to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.