Companies/Lightstep (ServiceNow Cloud Observability)/Incidents

Lightstep (ServiceNow Cloud Observability): Ingestion and web UI outage

Affected components: Observability - Data Ingestion - Trace StatisticsUS - Trace StatisticsObservability - Third-Party Notification Methods - Slack MessagingSlack MessagingObservability - Third-Party Notification Methods - Slack ConnectionsSlack ConnectionsObservability - Web UI - DashboardsUS - DashboardsObservability - APIUS - Observability - APIObservability - Data Ingestion - MetricsUS - MetricsObservability - Web UI - Service DirectoryUS - Service DirectoryObservability - Web UI - NotebooksUS - NotebooksObservability - Third-Party Notification Methods - Slack Apps/IntegrationsSlack Apps/IntegrationsObservability - Third-Party Notification Methods - PagerDuty Events APIPagerDuty Events APIObservability - Data Ingestion - Trace AssemblyUS - Trace AssemblyObservability - Web UI - ExplorerUS - ExplorerObservability - Alerting - Metric alertingUS - Unified AlertingObservability - Alerting - Trace alertingUS - Streams AlertingObservability - Web UI - Change IntelligenceUS - Change Intelligence

Timeline of updates

1. investigating2022-07-21 19:35 UTC

   We are currently investigating this issue.

2. investigating2022-07-21 19:35 UTC

   We are continuing to investigate this issue.

3. investigating2022-07-21 19:48 UTC

   We are still actively investigating this issue, and will provide another update shortly.

4. investigating2022-07-21 20:05 UTC

   We have now identified the issue and are actively implementing mitigations.

5. identified2022-07-21 20:20 UTC

   We have mitigated issues with metric ingestion. Trace ingestion, assembly and alerting are still experiencing degraded performance for some customers.

6. monitoring2022-07-21 20:30 UTC

   We have resolved issues with trace ingestion, assembly and alerting and believe the issue has been resolved, but are actively monitoring.

7. resolved2022-07-21 21:02 UTC

   All components are operational.

8. postmortem2022-07-29 00:21 UTC

   ### Summary Lightstep UI and Data Ingest experienced an outage triggered by a planned database change on July 21st. This database change caused a service failure, which led to a cascading failure in dependent services. Action was taken to restore the service in a degraded mode, which recovered the Lightstep UI. A subsequent roll back of the database change restored all systems and services. ### Timeline 12:22 PM: A database change was run causing a service to become unavailable. 12:25 PM: Cascading failures impact Lightstep UI and ingestion. Ingested data loss begins. 12:35 PM: Incident declared in status page. 12:49 PM: Root cause identified, service brought back in a degraded mode, recovery begins. 13:01 PM: Several systems stabilize, Lightstep UI recovers. 13:17 PM: Database change rolled back. Remaining affected services begin recovering. 13:30 PM: Ingestion recovers. Incident resolved, status page updated to “mitigated and monitoring”. 14:02 PM: Status page updated to fully operational. ### Action Items * Adding automated checks on code changes with database changes to help prevent the original root cause * Enabling the directly affected service to continue to serve traffic in a degraded state, rather than failing entirely. * Making architectural changes to Lightstep’s Data Ingest path to prevent data loss in these situations by continuing to accept and buffer incoming traffic.

Changes Rewindex caught on this incident

• Jul 21, 2022Major incident: Ingestion and web UI outage

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Lightstep (ServiceNow Cloud Observability) profileRecent Lightstep (ServiceNow Cloud Observability) incidentsMarkdownJSON