Companies / Wasabi / incidents

Wasabi: Network issues affecting all regions

Affected components: AP-Southeast-2 (Sydney)EU-Central-2 (Frankfurt)AP-Southeast-1 (Singapore)US-East-2 (N. Virginia)AP-Northeast-1 (Tokyo)US-Central-1 (Texas)Wasabi Management ConsoleEU-West-3 (London)EU-West-2 (Paris)EU-West-1 (London)Wasabi Account Control Manager ConsoleEU-South-1 (Milan)Wasabi Account Control APIEU-Central-1 (Amsterdam)AP-Northeast-2 (Osaka)CA-Central-1 (Toronto)US-West-1 (Oregon)US-East-1 (N. Virginia)

Timeline of updates

1. investigating2024-11-30 00:20 UTC

   We are currently investigating reported network errors across all regions. Access to both Console and S3 services may return errors. We will update this page as we have more information.

2. investigating2024-11-30 03:08 UTC

   We continue to investigate the issue. Some traffic continues to receive errors.

3. monitoring2024-11-30 03:43 UTC

   We have isolated the issue and resolution is underway. We expect all regions except us-east-1 to be successful. There is still some level of error responses being seen in Ashburn and we continue to work on that. We will continue to monitor all regions and update their status as well as that of us-east-1 as we continue to make progress.

4. monitoring2024-11-30 04:45 UTC

   The system has been restored to fully operational in all regions. We will populate the Postmortem section of this incident with more complete details as soon as possible.

5. resolved2024-11-30 04:46 UTC

   This incident has been resolved.

6. postmortem2024-12-04 14:08 UTC

   On 30 November 2024 from 2024-11-30 00:17 UTC to approximately 2024-11-30 03:00 UTC, Wasabi experienced an issue where client connection attempts to Wasabi Cloud Storage and the Web Console were impacted across all storage regions, resulting in all API calls to be returned as HTTP 5XX errors to clients. The cause behind this service degradation was due to an error within the internal messaging queue service responsible for taking client requests and routing them to our global database cluster. The internal messaging queue service failed to appropriately route these client requests across all nodes.  Wasabi’s Engineering and Operations teams was able to mitigate this issue by manually configuring internal servers to route requests across multiple database instances, allowing the system to recover and respond to requests appropriately. Once this action was taken, which restored service, the teams then worked to correct the root cause by working to recover the internal messaging queue and resume the automated task of proper client request handling.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Wasabi profile · Recent Wasabi incidents · Markdown · JSON