ChangesInfluxData (InfluxDB Cloud)Degraded query performance in GCP us-central1
InfluxData (InfluxDB Cloud): Major incident: Degraded query performance in GCP us-central1
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what InfluxData (InfluxDB Cloud) removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Degraded query performance in GCP us-central1
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-11-10 21:04 UTC
Before: not present
After: 2025-11-10 23:52 UTC
Before: not present
After: API Queries, Cloud Serverless: GCP - API Queries, Cloud Serverless: GCP - Tasks, Tasks
Before: not present
After: We are investigating increased query errors within the region
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: RCA for Cloud 2 prod01-us-central-1 query outage on Nov 10, 2025
# Summary
The SRE team has been working on a long-term project to review and rebalance the storage workloads, to provide better separation of workloads within nodepools, which will improve performance, reliability and ease of maintenance.
The Cloud 2 service is designed with a set of primary and secondary storage pods. During normal deployments, whenever a primary pod is restarted, the secondary pod takes over, so that the service is not interrupted. The change that was made in this cluster was a change to the primary and the secondary pods. When the configuration change was made in prod01-us-central-1, it caused some of the primary storage pods and their corresponding secondary pods to be restarted at the same time. While the pods were unavailable, queries failed. Once the pods restarted, the query error rate went back to normal levels and the cluster recovered.
# Timeline
**Time \(UTC\) What happened**
**8:45pm** The engineer-on-call was paged due to a high rate of query errors in the cluster. Upon investigation, the primary and secondary pods for the same storage slice are unavailable.
**8:50pm** Engineer-on-call identified that the problem was due to a misconfiguration that caused the primary and secondary pods to be restarted at the same time.
**9:00pm** The system will recover by itself when the pods restart with the new configuration, so no intervention is required.
**9:20pm** Queriers start recovering, and error rate is returning to pre-incident level. Some lingering errors in nginx.
**10:00pm** Query backlog is fully cleared. The cluster has fully recovered.
# Future Mitigations
**1. Reviewing procedures relating to storage pod reconfigurations**
Going forward we will ensure that configuration changes for primary pods and secondary pods are never applied at the same time.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Monday, November 10, 2025
1 incident change
- MajorMajor incident
Degraded query performance in GCP us-central1this change
All InfluxData (InfluxDB Cloud) changesInfluxData (InfluxDB Cloud) profileMarkdownJSON