Crusoe: Facility issue in us-east2-a
Affected components: GPU Virtual Machines - us-east2us-east2Storage - Shared DisksDatacenter NetworkingNetworking - Datacenter Networking
Timeline of updates
- investigating
We’re investigating an issue at one of our data center facilities that is affecting a subset of compute instances in us-east2-a. Some instances are currently unavailable. As a safety precaution, we’re powering down some additional hardware in the affected area. Instances on that hardware will become unavailable during this time. Our teams are actively working on this, and we’ll post another update within 30 minutes.
- investigating
As a safety precaution, we’ve shut off power to the affected hardware in us-east2-a. Instances running on that hardware are now powered off. Power won’t be restored until the area has been confirmed safe. Our teams are working on recovery, and we’ll post another update within 90 minutes.
- identified
We’ve started restoring power to part of the affected hardware in us-east2-a, and some instances will begin to come back online. The rest of the affected hardware will stay powered off while the facility issue is being remediated. Our teams are actively working on recovery, and we’ll post updates as events warrant.
- identified
Recovery work for the facility issue in us-east2-a is ongoing. The remaining affected hardware will stay powered off until required facility work and safety inspections are complete. We don’t have an estimated restoration time yet, and we’ll share one as soon as those inspections are done. In the meantime, we’re restoring affected instances on other available capacity in us-east2-a where possible, and a number of instances are already back online. If your instances are affected, our customer success team is in contact with you directly. We’ll post another update by 23:00 UTC
- identified
Work to address the facility issue in us-east2-a continues to progress. The hardware that's still affected will stay offline until facility repairs and safety checks are finished, and we don't yet have a time for full restoration. Our teams are still moving impacted instances onto other available capacity in us-east2-a where we can. We will post updates on further developments within the next 24 hours.
- identified
We have restored part of the affected capacity in us-east2-a and continue to move impacted instances onto other available capacity where we can. The remaining affected hardware will stay offline until facility repairs and safety checks are finished, and we don’t yet have a time for full restoration. We will post an update in 24 hours, or as events warrant.
- identified
Work to address the facility issue in us-east2-a continues. The affected hardware will stay offline until facility repairs and safety checks are complete, and it will be brought back online as that work allows. We don’t yet have a time for full restoration. Our teams continue to move impacted instances onto other available capacity in us-east2-a where we can. We will post an update in 24 hours, or as events warrant.
- identified
We have restored additional capacity in us-east2-a and are validating each host before returning it to service. Some affected customers are now fully restored, and we continue to return capacity to the remaining impacted customers. The remaining affected hardware will stay offline until facility repairs and safety checks are finished, and we don’t yet have a time for full restoration. We will post an update in 24 hours, or as events warrant.
- identified
Most affected instances in our Ohio region are back in service. A small number are still unavailable while we finish hardware checks and the last repairs. We’ll keep you posted on further developments
- resolved
This incident has been resolved.
Changes Rewindex caught on this incident
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).