Changes / CockroachDB / Failures on cluster operations
CockroachDB: Major incident: Failures on cluster operations
Summary
CockroachDB reported a major incident affecting AWS cluster operations, including creation and modification, starting 2023-12-22 02:01 UTC and lasting 24 h 45 min. Impact later narrowed to vertical scaling failures and delays. Resolved at 2023-12-23 02:46 UTC, the vendor stated all cluster operations should succeed and monitoring continued.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Failures on cluster operations
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2023-12-22 02:01 UTC
Before: not present
After: 2023-12-23 02:46 UTC
Before: not present
After: We are currently seeing some operations on AWS clusters failing, including cluster creation and modification. We've investigated and found a root cause and are working on a mitigation.
Before: not present
After: We've identified a root cause and are working on a solution. Current impact is limited to vertical scaling of AWS clusters only.
Before: not present
After: Vertical scaling of clusters on AWS may take longer than expected.
Before: not present
After: All cluster operations should now succeed. We are monitoring for any additional failures.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Failures on cluster operations (this change) CockroachDB reported a major incident affecting AWS cluster operations, including creation and modification, starting 2023-12-22 02:01 UTC and lasting 24 h 45 min. Impact later narrowed to vertical scaling failures and delays. Resolved at 2023-12-23 02:46 UTC, the vendor stated all cluster operations should succeed and monitoring continued. | Major incident |
All CockroachDB changes → · CockroachDB profile · Markdown · JSON