Changes / Omni Analytics / Omni pages not accessible
Omni Analytics: Critical incident: Omni pages not accessible
Summary
Omni Analytics pages in EastUsa were inaccessible during a critical incident from 2025-09-08 17:57 UTC to 2025-09-08 19:15 UTC, lasting 1 h 18 min. A recent code change generated unexpectedly large database queries that saturated CPU resources, causing query timeouts and connection failures; Omni reverted the change and resolved the incident.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Omni pages not accessible
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2025-09-08 17:57 UTC
Before: not present
After: 2025-09-08 19:15 UTC
Before: not present
After: EastUsa
Before: not present
After: Our monitoring picked up a spike in errors and we have begun getting reports from users. The team is investigating on highest priority
Before: not present
After: We have identified the source of the issue and are in the process of rapidly reverting a change to resolve.
Before: not present
After: We have completed rolling out the mitigation for this incident and are now in the monitoring phase.
Before: not present
After: We are now considering this incident resolved. A recent code change generated unexpectedly large database queries that saturated CPU resources, leading to query timeouts and subsequent connection failures. We initiated a rapid rollback once the cause was identified. Since mitigating we have reverted the change and are working to prevent similar issues in the future.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Omni pages not accessible (this change) Omni Analytics pages in EastUsa were inaccessible during a critical incident from 2025-09-08 17:57 UTC to 2025-09-08 19:15 UTC, lasting 1 h 18 min. A recent code change generated unexpectedly large database queries that saturated CPU resources, causing query timeouts and connection failures; Omni reverted the change and resolved the incident. | Major incident |
All Omni Analytics changes → · Omni Analytics profile · Markdown · JSON