Changes / CockroachDB / Serverless Cluster Resource Limit Throttling

CockroachDB: Major incident: Serverless Cluster Resource Limit Throttling

Summary

CockroachDB reported a major Serverless degradation in AWS eu-west-1 and AWS ap-south-1: some clusters remained throttled after resource limits were increased. It started at 2023-08-02 16:05 UTC and was resolved at 2023-08-02 19:44 UTC, lasting 3 h 39 min; previously throttled clusters now reflect resource-limit changes.

Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.

Major incidentLabel: degradation (model (Jev), 96%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Serverless Cluster Resource Limit Throttling
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2023-08-02 16:05 UTC
AddedResolved time#5
Before: not present
After: 2023-08-02 19:44 UTC
AddedAffected components#6
Before: not present
After: CockroachDB Serverless
AddedUpdateinvestigating#7
Before: not present
After: Some Serverless clusters that have hit their resource limits are still being throttled even after limits have been increased. Currently only AWS eu-west-1 and ap-south-1 host clusters are impacted
AddedUpdateinvestigating#8
Before: not present
After: Some Serverless clusters that have hit their resource limits are still being throttled even after limits have been increased.
AddedUpdateidentified#9
Before: not present
After: The scope of the impact has been limited to serverless clusters running AWS eu-west-1 and AWS ap-south-1. Clusters that have hit their resource limits will be throttled despite having their limits increased. One of two issues identified has been mitigated.
AddedUpdateresolved#10
Before: not present
After: This issue has now been resolved. Users with clusters that were previously throttled will now see their changes to resource limits reflected.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Serverless Cluster Resource Limit Throttling (this change)
CockroachDB reported a major Serverless degradation in AWS eu-west-1 and AWS ap-south-1: some clusters remained throttled after resource limits were increased. It started at 2023-08-02 16:05 UTC and was resolved at 2023-08-02 19:44 UTC, lasting 3 h 39 min; previously throttled clusters now reflect resource-limit changes.
Major incident

All CockroachDB changes → · CockroachDB profile · Markdown · JSON