# CockroachDB: Major incident: Failures on cluster operations

**Summary:** CockroachDB reported a major incident affecting AWS cluster operations, including creation and modification, starting 2023-12-22 02:01 UTC and lasting 24 h 45 min. Impact later narrowed to vertical scaling failures and delays. Resolved at 2023-12-23 02:46 UTC, the vendor stated all cluster operations should succeed and monitoring continued. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2023-12-22 02:01 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/0z8gzk7lp1bf](https://stspg.io/0z8gzk7lp1bf)
- [Company: CockroachDB](https://rewindex.com/company/cockroachdb.md)
- Versions: before (none: first archived) → after incident\_version:138/7jdp3zd08knb@89617ccffbfd
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Failures on cluster operations
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2023-12-22 02:01 UTC
```

**5. Added: Resolved time**

```diff
+ 2023-12-23 02:46 UTC
```

**6. Added: Update** (identified) · 2023-12-22 02:01 UTC

```diff
+ We are currently seeing some operations on AWS clusters failing, including cluster creation and modification. We've investigated and found a root cause and are working on a mitigation.
```

**7. Added: Update** (identified) · 2023-12-22 17:07 UTC

```diff
+ We've identified a root cause and are working on a solution. Current impact is limited to vertical scaling of AWS clusters only.
```

**8. Added: Update** (monitoring) · 2023-12-22 22:27 UTC

```diff
+ Vertical scaling of clusters on AWS may take longer than expected.
```

**9. Added: Update** (resolved) · 2023-12-23 02:46 UTC

```diff
+ All cluster operations should now succeed. We are monitoring for any additional failures.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2023-12-22 02:01 UTC | Major incident: Failures on cluster operations (this change) | outage |

All CockroachDB changes: [timeline](https://rewindex.com/company/cockroachdb/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ic9751b0e77be33 · Markdown: https://rewindex.com/change/ic9751b0e77be33.md · JSON: https://rewindex.com/api/v1/changes/ic9751b0e77be33 · Generated 2026-10-18T02:54:56.970Z