Changes / Omni Analytics / Slow response times in US Region

Omni Analytics: Major incident: Slow response times in US Region

Summary

Summary pending. The diff below shows exactly what changed.

Major incidentLabel: degradation (model (Jev), 99%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: Slow response times in US Region
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2025-04-29 16:39 UTC
AddedResolved time#5
Before: not present
After: 2025-04-29 22:10 UTC
AddedAffected components#6
Before: not present
After: EastUsa
AddedUpdateinvestigating#7
Before: not present
After: Users are experiencing slow responses in the US region, our team is investigating.
AddedUpdatemonitoring#8
Before: not present
After: We have mitigated the issue and are monitoring the system.
AddedUpdatemonitoring#9
Before: not present
After: We have identified the root cause of the issue and are implementing a fix.
AddedUpdateresolved#10
Before: not present
After: A model validation related process triggered repeated calls to an inefficient library function that exhausted memory in a subset of our US East infrastructure. A fix has been deployed that removes calls to the inefficient library function and we have validated it resolves the memory exhaustion.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Major incident: Slow response times in US Region (this change)Major incident

All Omni Analytics changes → · Omni Analytics profile · Markdown · JSON