Companies / Omni Analytics / incidents
Omni Analytics: Slow Queries
Omni Analytics reported major query degradation in EastUsa from 2024-04-08 16:55 UTC to 2024-04-08 17:05 UTC, lasting 10 min. Queries were slow or delayed because an information-schema query spike after cache expiration triggered a thundering herd, compounded by missing autoscaling logic; the incident was resolved, with autoscaling improvements planned.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: EastUsa
Timeline of updates
- investigating
Queries in the EastUsa region are slow and/or delayed.
- resolved
For approximately 10 minutes starting around 9:55am US/Pacific time, queries in the EastUsa region ran slower than normal. A spike of queries against an information schema where the cache had expired triggered a thundering herd effect. That combined with some missing logic in our auto scaling caused queues to back up. We will be working to address the issue in the auto scaling logic in the next few days and will also work to change our caching logic to avoid any further thundering herd effects.
Rewindex changes for this incident
- Major incident: Slow Queries 2024-04-08 16:55 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Omni Analytics profile · Recent Omni Analytics incidents · Markdown · JSON