# Wasabi: Network issues affecting all regions

- Impact (vendor’s label): none
- Status: postmortem
- Started: 2024-11-30 00:20 UTC · Resolved: 2024-11-30 04:46 UTC · Duration: 4.4 h
- Updates: 6 · Affected components: AP-Southeast-2 (Sydney), EU-Central-2 (Frankfurt), AP-Southeast-1 (Singapore), US-East-2 (N. Virginia), AP-Northeast-1 (Tokyo), US-Central-1 (Texas), Wasabi Management Console, EU-West-3 (London), EU-West-2 (Paris), EU-West-1 (London), Wasabi Account Control Manager Console, EU-South-1 (Milan), Wasabi Account Control API, EU-Central-1 (Amsterdam), AP-Northeast-2 (Osaka), CA-Central-1 (Toronto), US-West-1 (Oregon), US-East-1 (N. Virginia)
- [Vendor status page: https://stspg.io/69414c0dzw4g](https://stspg.io/69414c0dzw4g)
- [Company: Wasabi](https://rewindex.com/company/wasabi.md)

## Timeline of updates

- **investigating** 2024-11-30 00:20 UTC: We are currently investigating reported network errors across all regions. Access to both Console and S3 services may return errors. We will update this page as we have more information.
- **investigating** 2024-11-30 03:08 UTC: We continue to investigate the issue.  Some traffic continues to receive errors.
- **monitoring** 2024-11-30 03:43 UTC: We have isolated the issue and resolution is underway. We expect all regions except us-east-1 to be successful. There is still some level of error responses being seen in Ashburn and we continue to work on that.  We will continue to monitor all regions and update their status as well as that of us-east-1 as we continue to make progress.
- **monitoring** 2024-11-30 04:45 UTC: The system has been restored to fully operational in all regions. We will populate the Postmortem section of this incident with more complete details as soon as possible.
- **resolved** 2024-11-30 04:46 UTC: This incident has been resolved.
- **postmortem** 2024-12-04 14:08 UTC: On 30 November 2024 from 2024-11-30 00:17 UTC to approximately 2024-11-30 03:00 UTC, Wasabi experienced an issue where client connection attempts to Wasabi Cloud Storage and the Web Console were impacted across all storage regions, resulting in all API calls to be returned as HTTP 5XX errors to clients. The cause behind this service degradation was due to an error within the internal messaging queue service responsible for taking client requests and routing them to our global database cluster. The internal messaging queue service failed to appropriately route these client requests across all nodes.  Wasabi’s Engineering and Operations teams was able to mitigate this issue by manually configuring internal servers to route requests across multiple database instances, allowing the system to recover and respond to requests appropriately. Once this action was taken, which restored service, the teams then worked to correct the root cause by working to recover the internal messaging queue and resume the automated task of proper client request handling.


---
Canonical: https://rewindex.com/company/wasabi/incidents/dcb131s9f3sy · Markdown: https://rewindex.com/company/wasabi/incidents/dcb131s9f3sy.md · JSON: https://rewindex.com/api/v1/companies/wasabi/incidents/dcb131s9f3sy · Generated 2026-10-18T09:40:28.035Z