Rewindex

CompaniesCrusoeIncidents

Crusoe: VM creation and networking failure for A100 Infiniband type VMs in us-east region

MajorStatus: resolvedStarted Resolved (3.0 h)4 updatesstspg.io (opens in new tab)

Affected components: GPU Virtual Machines - us-east1us-east1

Timeline of updates

  1. investigating

    We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

  2. identified

    The issue has been identified, and we have tested a fix internally. We are working on rolling out the fix to our A100 Infiniband type servers now.

  3. monitoring

    A fix has been implemented, and we are monitoring the environment for now.

  4. resolved

    This incident is now resolved

Changes Rewindex caught on this incident

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Crusoe profileRecent Crusoe incidentsMarkdownJSON