ChangesInfluxData (InfluxDB Cloud)Elevated Query Error Rate
InfluxData (InfluxDB Cloud): Major incident: Elevated Query Error Rate
Summary
Summary pending. The diff below shows exactly what changed.
Incident page: the full timeline of updates
What changed
Struck in red is what InfluxData (InfluxDB Cloud) removed; blue is what they added. Muted text is unchanged context.
Before: not present
After: Elevated Query Error Rate
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-06-13 16:13 UTC
Before: not present
After: 2025-06-13 22:14 UTC
Before: not present
After: API Queries, Cloud Serverless: Azure, W. Europe - API Queries, Cloud Serverless: Azure, W. Europe - Tasks, Tasks
Before: not present
After: We have observed an elevated query error rate and are investigating.
Before: not present
After: The issue has been identified and a fix is being put into place.
Before: not present
After: A fix has been implemented and deployed. We are monitoring at this time.
Before: not present
After: We are continuing to closely monitor the health of the query API
Before: not present
After: We are continuing to monitor this region for any further degradation
Before: not present
After: This incident has been resolved.
Before: not present
After: In a multi-tenant environment, such as Cloud 2, there can be unpredictable changes in workload, as different customers on the shared platform may increase or decrease their activity on the cluster with no advance notice. While we try to shelter customers from one another, via rate limits and some partitioning, there is no way to fully protect customers from the impact of a "noisy neighbor". We maintain excess capacity in the cluster to provide some amount of headroom for increased workloads, and we also have rate limits to ensure that one customer does not overwhelm the cluster \(via writes, deletes or queries\), but the rate limits are not very fine-grained, and at times, a single customer can stress the cluster. When that occurs, the team is automatically notified, and we add additional resources to accommodate the extra workload. It can sometimes take time for the extra capacity to be fully deployed, leading to temporary performance problems. This is what occurred in the Azure eu-west cluster on June 13th. We were notified that query performance was degraded and the team increased the storage pods to provide extra capacity. Once the storage pods limits were fully applied, the query TTBR recovered.
We apologize for the inconvenience.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
Friday, June 13, 2025
1 incident change
- MajorMajor incident
Elevated Query Error Ratethis change
All InfluxData (InfluxDB Cloud) changesInfluxData (InfluxDB Cloud) profileMarkdownJSON