ChangesCrusoeService Degradation in us-east1-a Region due to Power Disruption
Crusoe: Major incident: Service Degradation in us-east1-a Region due to Power Disruption
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Crusoe removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Service Degradation in us-east1-a Region due to Power Disruption
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-08-20 17:09 UTC
Before: not present
After: 2025-08-21 23:35 UTC
Before: not present
After: GPU Virtual Machines - us-east1, Infiniband Networks, Networking - Infiniband Networks, us-east1
Before: not present
After: We're investigating a service degradation in our us-east1-a region, triggered by a facility power disruption at our data center.
The primary impact is to the Infiniband networking fabric, which may cause intermittent errors or failures for multi-node, distributed workloads. Some customers may also experience individual virtual machines becoming unavailable. Our teams are working to identify all affected resources.
Our engineering teams are actively working to stabilize the affected systems and mitigate the risk of further disruption. We are coordinating with our data center provider to support their remediation efforts and restore full service resiliency as quickly as possible. We apologize for any impact this is causing.
Before: not present
After: We are continuing to investigate and mitigate the service degradation affecting our us-east1-a region, following a facility power disruption at our data center. Our teams remain in close coordination with the data center provider as they work to fully restore services. Recovery of critical systems remains our top priority.
We sincerely apologize for the ongoing impact and appreciate your continued patience as we work to resolve the issue.
Before: not present
After: We have successfully mitigated the issue affecting the us-east1-a region. The facility power disruption has been addressed, and the impacted Infiniband networking fabric and associated systems have returned to normal operation.
All affected services are now stable, and full functionality has been restored. Our teams will continue to monitor the region closely to ensure continued stability.
We appreciate your patience during this incident and apologize again for any disruption it may have caused.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Wednesday, August 20, 2025
1 incident change
- MajorMajor incident
Service Degradation in us-east1-a Region due to Power Disruptionthis change