Companies / Omni Analytics / incidents

Omni Analytics: Slow response times in US Region

Impact: majorStatus: resolvedStarted Resolved (5.5 h)4 updatesSource: stspg.io (opens in new tab)
Summary

Omni Analytics reported a major degradation causing slow responses in the US region (EastUsa), from 2025-04-29 16:39 UTC to 2025-04-29 22:10 UTC, lasting 5 h 30 min. A model validation process repeatedly called an inefficient library function, exhausting memory in some US East infrastructure; the deployed fix removed those calls and was validated to resolve the memory exhaustion.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: EastUsa

Timeline of updates

  1. investigating

    Users are experiencing slow responses in the US region, our team is investigating.

  2. monitoring

    We have mitigated the issue and are monitoring the system.

  3. monitoring

    We have identified the root cause of the issue and are implementing a fix.

  4. resolved

    A model validation related process triggered repeated calls to an inefficient library function that exhausted memory in a subset of our US East infrastructure. A fix has been deployed that removes calls to the inefficient library function and we have validated it resolves the memory exhaustion.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Omni Analytics profile · Recent Omni Analytics incidents · Markdown · JSON