Wasabi: Wasabi Account Control Manager Console views for Control Accounts, Sub-Accounts and Invoices not loading

Impact: minorStatus: postmortemStarted 2024-10-30 20:14 UTCResolved 2024-10-31 09:55 UTC (14 h)5 updatesSource: stspg.io

Affected components: Wasabi Account Control Manager Console

Timeline of updates

investigating2024-10-30 20:14 UTC
We are currently investigating this issue and expect to have these views restored shortly.
investigating2024-10-31 00:34 UTC
While some of the views have been restored, we continue to work to resolve the issue. This page will be updated as more progress is made.
identified2024-10-31 00:44 UTC
The issue has been identified and a fix is being implemented.
resolved2024-10-31 09:55 UTC
This incident has been resolved.
postmortem2024-11-07 15:24 UTC
From 2024-10-31 00:42 UTC to 2024-10-31 10:03 UTC, we experienced an issue where Control Accounts, Sub-Accounts and Invoice data in WACM was not loading properly when attempted to be viewed by the client. Upon investigation by our Engineering Team, it was found that the nodes responsible for fetching and delivering this data were in a bad state. Once the nodes were restored to a functional state, our team started the process of re-indexing the data, which would allow the data to be served via the WACM portal successfully. At 10:03 UTC on October 31, 2024, the WACM data was fully available to customers.