ChangesCrusoeVM creation and networking failure for A100 Infiniband type VMs in us-east region
Crusoe: Major incident: VM creation and networking failure for A100 Infiniband type VMs in us-east region
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what Crusoe removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: VM creation and networking failure for A100 Infiniband type VMs in us-east region
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-08-01 03:18 UTC
Before: not present
After: 2025-08-01 06:21 UTC
Before: not present
After: GPU Virtual Machines - us-east1, us-east1
Before: not present
After: We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet.
Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online.
Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place.
Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.
Before: not present
After: The issue has been identified, and we have tested a fix internally. We are working on rolling out the fix to our A100 Infiniband type servers now.
Before: not present
After: A fix has been implemented, and we are monitoring the environment for now.
Before: not present
After: This incident is now resolved
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Friday, August 1, 2025
1 incident change
- MajorMajor incident
VM creation and networking failure for A100 Infiniband type VMs in us-east regionthis change