LogRocket: Performance issues with primary search database

Impact: majorStatus: resolvedStarted 2023-01-18 15:41 UTCResolved 2023-01-20 14:00 UTC (46 h)9 updatesSource: stspg.io

Summary

LogRocket reported a major primary search database degradation affecting session searches, metric charts, issues and alerting; session data collection was unaffected. It started at 2023-01-18 15:41 UTC and resolved at 2023-01-20 14:00 UTC, lasting 46 h 18 min. LogRocket applied a fix and processed event backlogs, reporting no further degradation after mitigation.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: DashboardMetricsAlertsIssues

Timeline of updates

investigating2023-01-18 15:41 UTC
We are currently investigating an issue impacting our primary search database. Searching for sessions, loading metric charts, issues and alerting are all impacted. Session data collection has not been impacted.
identified2023-01-18 16:24 UTC
We have identified the issue with our database vendor and are working with them to remedy the situation.
identified2023-01-18 18:00 UTC
We are continuing to work with our vendor to resolve the issue with ingesting new data.
identified2023-01-18 20:02 UTC
We are continuing to work with our vendor to resolve the issue.
identified2023-01-19 01:00 UTC
We are continuing to work with our vendor to remedy the situation. Session data collection continues to be unaffected, but secondary indexing that powers search and metrics continues to be delayed.
identified2023-01-19 05:03 UTC
We have restored some indexing to the database but are continuing to fully resolve the situation.
monitoring2023-01-19 08:15 UTC
We have identified the cause of the issue, applied a fix and have begun processing the backlog of events.
monitoring2023-01-19 15:41 UTC
We have completed processing of our primary indexing backlog, session filtering and metrics will be mostly up-to-date. Our secondary processing system is currently processing and includes some metrics, issues and alerting.
resolved2023-01-20 14:00 UTC
We've seen no additional performance degradation after our mitigation work.

Rewindex changes for this incident

Major incident: Performance issues with primary search database 2023-01-18 15:41 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). LogRocket profile · Recent LogRocket incidents · Markdown · JSON