Rewindex

ChangesCrusoeFacility issue in us-east2-a

Crusoe: Critical incident: Facility issue in us-east2-a

Major incidentcritical (vendor)Changed Detected stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what Crusoe removed; blue is what they added. Muted text is unchanged context.

AddedTitle#1
Before: not present
After: Facility issue in us-east2-a
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2026-10-15 13:12 UTC
AddedResolved time#5
Before: not present
After: 2026-10-18 20:22 UTC
AddedAffected components#6
Before: not present
After: Datacenter Networking, GPU Virtual Machines - us-east2, Networking - Datacenter Networking, Storage - Shared Disks, us-east2
AddedUpdateinvestigating#7
Before: not present
After: We’re investigating an issue at one of our data center facilities that is affecting a subset of compute instances in us-east2-a. Some instances are currently unavailable. As a safety precaution, we’re powering down some additional hardware in the affected area. Instances on that hardware will become unavailable during this time. Our teams are actively working on this, and we’ll post another update within 30 minutes.
AddedUpdateinvestigating#8
Before: not present
After: As a safety precaution, we’ve shut off power to the affected hardware in us-east2-a. Instances running on that hardware are now powered off. Power won’t be restored until the area has been confirmed safe. Our teams are working on recovery, and we’ll post another update within 90 minutes.
AddedUpdateidentified#9
Before: not present
After: We’ve started restoring power to part of the affected hardware in us-east2-a, and some instances will begin to come back online. The rest of the affected hardware will stay powered off while the facility issue is being remediated. Our teams are actively working on recovery, and we’ll post updates as events warrant.
AddedUpdateidentified#10
Before: not present
After: Recovery work for the facility issue in us-east2-a is ongoing. The remaining affected hardware will stay powered off until required facility work and safety inspections are complete. We don’t have an estimated restoration time yet, and we’ll share one as soon as those inspections are done. In the meantime, we’re restoring affected instances on other available capacity in us-east2-a where possible, and a number of instances are already back online. If your instances are affected, our customer success team is in contact with you directly. We’ll post another update by 23:00 UTC
AddedUpdateidentified#11
Before: not present
After: Work to address the facility issue in us-east2-a continues to progress. The hardware that's still affected will stay offline until facility repairs and safety checks are finished, and we don't yet have a time for full restoration. Our teams are still moving impacted instances onto other available capacity in us-east2-a where we can. We will post updates on further developments within the next 24 hours.
AddedUpdateidentified#12
Before: not present
After: We have restored part of the affected capacity in us-east2-a and continue to move impacted instances onto other available capacity where we can. The remaining affected hardware will stay offline until facility repairs and safety checks are finished, and we don’t yet have a time for full restoration. We will post an update in 24 hours, or as events warrant.
AddedUpdateidentified#13
Before: not present
After: Work to address the facility issue in us-east2-a continues. The affected hardware will stay offline until facility repairs and safety checks are complete, and it will be brought back online as that work allows. We don’t yet have a time for full restoration. Our teams continue to move impacted instances onto other available capacity in us-east2-a where we can. We will post an update in 24 hours, or as events warrant.
AddedUpdateidentified#14
Before: not present
After: We have restored additional capacity in us-east2-a and are validating each host before returning it to service. Some affected customers are now fully restored, and we continue to return capacity to the remaining impacted customers. The remaining affected hardware will stay offline until facility repairs and safety checks are finished, and we don’t yet have a time for full restoration. We will post an update in 24 hours, or as events warrant.
AddedUpdateidentified#15
Before: not present
After: Most affected instances in our Ohio region are back in service. A small number are still unavailable while we finish hardware checks and the last repairs. We’ll keep you posted on further developments
AddedUpdateresolved#16
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

Thursday, October 15

1 incident change

All Crusoe changesCrusoe profileMarkdownJSON