Companies / CockroachDB / incidents

CockroachDB: Failures on cluster operations

Impact: majorStatus: resolvedStarted Resolved (25 h)4 updatesSource: stspg.io (opens in new tab)
Summary

CockroachDB reported a major incident affecting AWS cluster operations, including creation and modification, starting 2023-12-22 02:01 UTC and lasting 24 h 45 min. Impact later narrowed to vertical scaling failures and delays. Resolved at 2023-12-23 02:46 UTC, the vendor stated all cluster operations should succeed and monitoring continued.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Timeline of updates

  1. identified

    We are currently seeing some operations on AWS clusters failing, including cluster creation and modification. We've investigated and found a root cause and are working on a mitigation.

  2. identified

    We've identified a root cause and are working on a solution. Current impact is limited to vertical scaling of AWS clusters only.

  3. monitoring

    Vertical scaling of clusters on AWS may take longer than expected.

  4. resolved

    All cluster operations should now succeed. We are monitoring for any additional failures.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). CockroachDB profile · Recent CockroachDB incidents · Markdown · JSON