Rewindex

CompaniesNebiusIncidents

Nebius: Region us-central1 partial degradation

MajorStatus: postmortemStarted Resolved (22 h)13 updatesstspg.io (opens in new tab)

Affected components: Object StorageUS-CENTRAL1 - Object StorageUS-CENTRAL1 - Virtual Private Cloud (Networking)Virtual Private Cloud (Networking)Token FactoryUS-CENTRAL1 - Token FactoryServerless AIUS-CENTRAL1 - Serverless AICompute CloudUS-CENTRAL1 - Compute CloudManaged Service for MLflowUS-CENTRAL1 - Managed Service for MLflowManaged Service for SkyPilotUS-CENTRAL1 - Managed Service for SkyPilotManaged Service for Kubernetes®US-CENTRAL1 - Managed Service for Kubernetes®Managed Service for PostgreSQL®US-CENTRAL1 - Managed Service for PostgreSQL®

Timeline of updates

  1. investigating

    We are currently investigating this issue.

  2. identified

    The root cause has been identified as overheating hardware. GPU performance is impacted, and some nodes may be unavailable.

  3. identified

    We are still experiencing temperature issues in the us-central1. Team is working on it

  4. identified

    The root cause was fixed, and the situation is stabilizing. The services are recovering.

  5. identified

    Services are gradually continuing to recover.

  6. identified

    Services are still gradually continuing to recover. Some virtual machines are already running, but not all of them.

  7. identified

    Services are still gradually continuing to recover. Many of the virtual machines are already running, but not all of them.

  8. identified

    Most nodes are operating normally. The Managed Service for Kubernetes® is experiencing a partial outage, and the team is actively working to restore it.

  9. identified

    Most nodes are operating normally. The Managed Service for Kubernetes® is still degraded, but the situation is improving.

  10. identified

    The Managed Service for Kubernetes is nearly fully restored, though a subset of nodes remains unavailable.

  11. identified

    Most critical issues have been resolved and the region is operational, though a subset of nodes remains unavailable.

  12. resolved

    This incident has been resolved.

  13. postmortem

    [https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19](https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19)

Changes Rewindex caught on this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Nebius profileRecent Nebius incidentsMarkdownJSON