Changes / Omni Analytics / Degraded Performance

Omni Analytics: Major incident: Degraded Performance

Summary

Omni Analytics reported major degraded performance in EastUsa, with most queries failing to complete. The incident ran from 2024-04-09 18:44 UTC to 2024-04-10 01:04 UTC (6 h 19 min). Queries against a specific database type exhausted memory, repeatedly crashing querying until cancelled; Omni identified a fix for release that day and later marked the incident resolved.

Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 70%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Degraded Performance
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2024-04-09 18:44 UTC
AddedResolved time#5
Before: not present
After: 2024-04-10 01:04 UTC
AddedAffected components#6
Before: not present
After: EastUsa
AddedUpdateinvestigating#7
Before: not present
After: We are seeing elevated error rates. Investigating...
AddedUpdateinvestigating#8
Before: not present
After: We are continuing to investigate this issue.
AddedUpdatemonitoring#9
Before: not present
After: The application is currently stable and we believe we have identified the cause.
AddedUpdatemonitoring#10
Before: not present
After: From 11:39am to 11:52am Pacific Time the majority of queries run in Omni failed to complete. Our engineers have traced the issue back to a problematic set of queries against a specific database type that overflowed the memory available to our infrastructure, causing our querying system to crash repeatedly until the queries were cancelled. We have identified a fix that will be released today.
AddedUpdateresolved#11
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Degraded Performance (this change)
Omni Analytics reported major degraded performance in EastUsa, with most queries failing to complete. The incident ran from 2024-04-09 18:44 UTC to 2024-04-10 01:04 UTC (6 h 19 min). Queries against a specific database type exhausted memory, repeatedly crashing querying until cancelled; Omni identified a fix for release that day and later marked the incident resolved.
Major incident

All Omni Analytics changes → · Omni Analytics profile · Markdown · JSON