Changes / Wasabi / System Errors in US-WEST-1 Region

Wasabi: Major incident: System Errors in US-WEST-1 Region

Summary

Summary pending. The diff below shows exactly what changed.

Major incidentLabel: outage (model (Jev), 56%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: System Errors in US-WEST-1 Region
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2024-09-25 13:51 UTC
AddedResolved time#5
Before: not present
After: 2024-09-25 15:22 UTC
AddedAffected components#6
Before: not present
After: US-West-1 (Oregon)
AddedUpdateinvestigating#7
Before: not present
After: We are currently investigating system errors occurring in our US-WEST-1 region.
AddedUpdateinvestigating#8
Before: not present
After: We are continuing to investigate this issue.
AddedUpdateidentified#9
Before: not present
After: Our Operations Team has identified the issue and is working to correct the system errors.
AddedUpdatemonitoring#10
Before: not present
After: Our Operations Team has corrected the issue, and the US-WEST-1 region is back in an operational state. We will continue to monitor the region.
AddedUpdateinvestigating#11
Before: not present
After: We are continuing to investigate this issue.
AddedUpdateresolved#12
Before: not present
After: This incident is now resolved. Please reach out to support@wasabi.com if you continue to see any errors related to this incident.
AddedUpdatepostmortem#13
Before: not present
After: From 2024-09-25 13:49 UTC to 2024-09-25 13:54 UTC, an incident occurred in the Hillsboro region, resulting in performance degradation for customers and impacting their ability to make S3 API calls against buckets in the region.  During an upgrade to the region, the Operations team encountered an issue where a misconfiguration caused a service to falsely report as ready for operation. This resulted in service readiness failures and triggered failures within the region. To address the situation, the Operations Team corrected the misconfiguration and restarted the affected service.  By 13:54 UTC, the S3 service in the US-WEST-1 region was fully restored, and normal operations resumed.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Major incident: System Errors in US-WEST-1 Region (this change)Major incident

All Wasabi changes → · Wasabi profile · Markdown · JSON