# InfluxData (InfluxDB Cloud): Major incident: Query Performance Degradation in Azure West Europe **Summary:** _summary pending_ - Kind: Major incident · Label: major (vendor) - Changed: 2026-09-29 05:28 UTC · Detected: 2026-10-19 18:33 UTC - [Source: https://stspg.io/wl8wp3k886fx](https://stspg.io/wl8wp3k886fx) - [Company: InfluxData (InfluxDB Cloud)](https://rewindex.com/company/influxdata.md) - Versions: before (none: first archived) → after incident\_version:738/q52cftvv6fnl@dbcf8b973eca - Change 1 of 1 for this incident Incident page (full timeline): [Query Performance Degradation in Azure West Europe](https://rewindex.com/company/influxdata/incidents/q52cftvv6fnl.md) ## What changed **1. Added: Title** ```diff + Query Performance Degradation in Azure West Europe ``` **2. Added: Impact** ```diff + major ``` **3. Added: Status** ```diff + postmortem ``` **4. Added: Start time** ```diff + 2026-09-29 05:28 UTC ``` **5. Added: Resolved time** ```diff + 2026-10-02 18:17 UTC ``` **6. Added: Affected components** ```diff + API Queries, Cloud Serverless: Azure, W. Europe - API Queries ``` **7. Added: Update** (investigating) · 2026-09-29 05:28 UTC ```diff + We are currently investigating an issue affecting query performance in Azure West Europe. ``` **8. Added: Update** (investigating) · 2026-09-29 06:38 UTC ```diff + We are continuing to investigate this issue and will provide further updates as soon as available. ``` **9. Added: Update** (identified) · 2026-09-29 09:09 UTC ```diff + The issue has been identified and a fix is being implemented. ``` **10. Added: Update** (identified) · 2026-09-29 11:34 UTC ```diff + We are continuing to work on a fix for this issue. ``` **11. Added: Update** (identified) · 2026-09-29 15:25 UTC ```diff + We are continuing work to remediate immediate query performance degradation, and investigating actions for a permanent fix. ``` **12. Added: Update** (monitoring) · 2026-09-29 16:39 UTC ```diff + Query performance has returned to normal levels. Planned stabilisation work is still ongoing, and we are monitoring closely. ``` **13. Added: Update** (monitoring) · 2026-09-29 18:33 UTC ```diff + Query performance remains stable, and we continue to monitor closely. Additional mitigation and stabilization work is ongoing. ``` **14. Added: Update** (monitoring) · 2026-09-29 23:31 UTC ```diff + Query performance remains stable, and we continue to monitor closely as traffic increases. We will continue to assess performance and provide updates. ``` **15. Added: Update** (monitoring) · 2026-09-30 07:34 UTC ```diff + We are working on resolving the ongoing intermittent query performance. During peak usage periods, large or complex queries may run slower than normal. Writes and data durability are not affected at this time. We have deployed several improvements overnight and further changes are in progress. ``` **16. Added: Update** (identified) · 2026-09-30 07:53 UTC ```diff + We have identified the issue and the team is actively working on implementing a fix. ``` **17. Added: Update** (investigating) · 2026-09-30 08:50 UTC ```diff + We are still having issues in the region, the engineering team is working on it. ``` **18. Added: Update** (investigating) · 2026-09-30 10:49 UTC ```diff + We are continuing to investigate this issue. ``` **19. Added: Update** (investigating) · 2026-09-30 12:08 UTC ```diff + We are continuing to investigate this issue. ``` **20. Added: Update** (investigating) · 2026-09-30 13:44 UTC ```diff + We have made several recent changes to improve query performance and we continue to monitor the region closely. ``` **21. Added: Update** (monitoring) · 2026-09-30 14:24 UTC ```diff + Performance has returned to normal levels following an update. We will continue to monitor. ``` **22. Added: Update** (monitoring) · 2026-09-30 17:16 UTC ```diff + We have increased isolation in the storage layer for the impacted region to provide greater amounts of resilience and to allow for faster recovery. We are continuing to investigate heightened memory consumption and page faults within the storage layer. Query success rates and response durations have returned to normal levels. We will continue work to increase the resiliency of this region over the next 24 hours and will carefully monitor for any increase in query latency. ``` **23. Added: Update** (monitoring) · 2026-10-01 08:13 UTC ```diff + The cluster remains stable, and we are continuing to monitor it. ``` **24. Added: Update** (resolved) · 2026-10-02 18:17 UTC ```diff + We appreciate the continued patience while we worked through this incident. We have been monitoring the previous fixes for the last 24 hours and all changes have responded well during sustained traffic. The region is stable and we are resolving this incident. An RCA is in progress and will be added to our status page as soon as its available. ``` **25. Added: Update** (postmortem) · 2026-10-06 19:00 UTC ```diff + # RCA # Query Performance Degradation in Azure West Europe, September 27 to October 1, 2026 ## Summary From the evening of September 27 to the morning of September 30 \(UTC\), customers in the Azure West Europe region experienced slow queries and timeouts. Writes were accepted throughout, but on some partitions, new data was not immediately queryable. Earlier incidents on this cluster \(September 7, 11, 15 and 16\) produced the same symptom with different triggers and have their own RCAs. We believe the condition described herein was the root cause of all prior incidents. ## Cause All data in the cluster is stored on redundant pods across many physical nodes. These pods read time-series data through memory-mapped files. The host’s operating system keeps the most frequently accessed parts of those files in a page cache built from memory not consumed by the pods themselves. In this region, each pod holds about 365 GB of data. The cache on a host could not hold the data its pods were querying, so reads that would normally come from cache became page faults served from disk. Each pod read about 90 MB/s from disk, thus a host with ten pods reached roughly 990 MB/s, which is near the throughput ceiling on each host. Memory consumption looked normal on every metric we monitor. The storage process itself uses little memory, and the cache takes the remainder, so a shortage would not appear as memory pressure or shortfall. At the throughput ceiling, index lookups on the write path were delayed by disk reads, causing writes to fall behind and pods to fail their replication-lag health check. Pods in this region spent 8.9% of the day past the lag threshold, against 0.09% in a comparable region. At peak, both replicas of the busiest partitions fell behind together, creating access delays which were most acutely visible through the query path. Slow disk caused health and readiness checks to time out, so healthy pods were removed from rotation and restarted. Each restart took 30 to 85 minutes to recover, during which the pod's partner carried the load alone, with the rebuild adding even more disk load. A file-reference safeguard was also causing 50\+ pod restarts a day where a normal rate would be 0. ## Remediation While early changes helped, and resolved many trouble indicators, they did not solve the root cause. We doubled query-layer capacity, restarted pods, raised the readiness timeouts, raised cleanup timeouts to avoid safeguard restarts, and doubled recovery concurrency after a restart. We ruled out a controller lock, a per-disk limit, and a single tenant's load. The root cause fix was to raise each storage pod's resource request so that each host holds fewer pods. Each pod now has twice as much memory for page cache and disk throughput. Within 1 hour of this change, query abandonment fell from 31% to 0.1%. Page faults and disk reads dropped sharply. During the next morning's peak, abandonment was 0.06%, within normal limits. ## Timeline \(UTC\) Sep 27 22:58 - Query queue depth alert fires; incident opened. Query-layer capacity increased over the next hours without lasting effect Sep 28 07:00 - Degradation at morning peak; storage pods fall behind on writes and fail health checks; stuck pods restarted one at a time Sep 28 11:33 - Restart loop identified: slow disk causes health checks to trigger restart; each restart takes 30 to 85 minutes to recover Sep 28 13:45 - Lost file-reference cleanup identified as a second source of forced restarts Sep 29 08:58 - Readiness probe timeout raised and rolled out fleet-wide Sep 29 11:10 - Throughput appears to recover; degradation returns, with 36% of queries abandoned Sep 29 16:20 - Queue clears after recovery-concurrency and probe changes Sep 30 00:25 - Cleanup timeout raised to 24 hours; forced restarts stop Sep 30 07:00 - Degradation at peak; abandonment about 55% Sep 30 11:59 - Analysis shows cache capacity per pod is too small and disk reads are at the host ceiling Sep 30 12:43 - Rollout begins to spread pods across more nodes Sep 30 13:21 - Query performance returns to normal; abandonment down to 0.1% within the hour Sep 30 14:20 - Page faults and disk reads confirmed sharply lower after rollout Oct 1 07:00 to 09:00 - Morning peak with 0.06% abandonment versus 40% to 55% the day before Oct 1 09:48 - Incident declared resolved internally ## Future mitigation We added warning alerts for page-fault rate, cache coverage, disk reads on storage volumes, and query abandonment. Each alert is designed so that cache pressure appears before causing external visibility. Runbooks now suggest major page faults as a leading indicator of storage performance degradation, with details on “relief valves” available to tune. We are continuing to track down the storage-engine reference leak and moving the disk check out of the health path. Neither leak nor check is a risk factor for triggering this type of incident; they are conditions observed during diagnosis and remediation that warrant investigation. ``` ## History of this incident | When (UTC) | Change | Label | | --- | --- | --- | | 2026-09-29 05:28 UTC | Major incident: Query Performance Degradation in Azure West Europe (this change) | major | All InfluxData (InfluxDB Cloud) changes: [timeline](https://rewindex.com/company/influxdata/changes.md) _Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._ --- Canonical: https://rewindex.com/change/i9e3de5429ce6eb · Markdown: https://rewindex.com/change/i9e3de5429ce6eb.md · JSON: https://rewindex.com/api/v1/changes/i9e3de5429ce6eb · Generated 2026-10-19T23:13:37.219Z