# Lightstep (ServiceNow Cloud Observability): Ingestion and web UI outage

- Impact (vendor’s label): major
- Status: postmortem
- Started: 2022-07-21 20:05 UTC · Resolved: 2022-07-21 21:02 UTC · Duration: 57 min
- Updates: 8 · Affected components: Observability - Data Ingestion - Trace Statistics, US - Trace Statistics, Observability - Third-Party Notification Methods - Slack Messaging, Slack Messaging, Observability - Third-Party Notification Methods - Slack Connections, Slack Connections, Observability - Web UI - Dashboards, US - Dashboards, Observability - API, US - Observability - API, Observability - Data Ingestion - Metrics, US - Metrics, Observability - Web UI - Service Directory, US - Service Directory, Observability - Web UI - Notebooks, US - Notebooks, Observability - Third-Party Notification Methods - Slack Apps/Integrations, Slack Apps/Integrations, Observability - Third-Party Notification Methods - PagerDuty Events API, PagerDuty Events API, Observability - Data Ingestion - Trace Assembly, US - Trace Assembly, Observability - Web UI - Explorer, US - Explorer, Observability - Alerting - Metric alerting, US - Unified Alerting, Observability - Alerting - Trace alerting, US - Streams Alerting, Observability - Web UI - Change Intelligence, US - Change Intelligence
- [Vendor status page: https://stspg.io/zdcb2q1tz85w](https://stspg.io/zdcb2q1tz85w)
- [Company: Lightstep (ServiceNow Cloud Observability)](https://rewindex.com/company/lightstep.md)

## Timeline of updates

- **investigating** 2022-07-21 19:35 UTC: We are currently investigating this issue.
- **investigating** 2022-07-21 19:35 UTC: We are continuing to investigate this issue.
- **investigating** 2022-07-21 19:48 UTC: We are still actively investigating this issue, and will provide another update shortly.
- **investigating** 2022-07-21 20:05 UTC: We have now identified the issue and are actively implementing mitigations.
- **identified** 2022-07-21 20:20 UTC: We have mitigated issues with metric ingestion.  Trace ingestion, assembly and alerting are still experiencing degraded performance for some customers.
- **monitoring** 2022-07-21 20:30 UTC: We have resolved issues with trace ingestion, assembly and alerting and believe the issue has been resolved, but are actively monitoring.
- **resolved** 2022-07-21 21:02 UTC: All components are operational.
- **postmortem** 2022-07-29 00:21 UTC: \#\#\# Summary Lightstep UI and Data Ingest experienced an outage triggered by a planned database change on July 21st. This database change caused a service failure, which led to a cascading failure in dependent services. Action was taken to restore the service in a degraded mode, which recovered the Lightstep UI. A subsequent roll back of the database change restored all systems and services. \#\#\# Timeline 12:22 PM: A database change was run causing a service to become unavailable. 12:25 PM: Cascading failures impact Lightstep UI and ingestion. Ingested data loss begins. 12:35 PM: Incident declared in status page. 12:49 PM: Root cause identified, service brought back in a degraded mode, recovery begins. 13:01 PM: Several systems stabilize, Lightstep UI recovers. 13:17 PM: Database change rolled back. Remaining affected services begin recovering. 13:30 PM: Ingestion recovers. Incident resolved, status page updated to “mitigated and monitoring”. 14:02 PM: Status page updated to fully operational. \#\#\# Action Items \* Adding automated checks on code changes with database changes to help prevent the original root cause \* Enabling the directly affected service to continue to serve traffic in a degraded state, rather than failing entirely. \* Making architectural changes to Lightstep’s Data Ingest path to prevent data loss in these situations by continuing to accept and buffer incoming traffic.

## Rewindex changes for this incident

- 2022-07-21 20:05 UTC: [Major incident: Ingestion and web UI outage](https://rewindex.com/change/ia8f10b64aef95c.md)


---
Canonical: https://rewindex.com/company/lightstep/incidents/jyj858ftvg2p · Markdown: https://rewindex.com/company/lightstep/incidents/jyj858ftvg2p.md · JSON: https://rewindex.com/api/v1/companies/lightstep/incidents/jyj858ftvg2p · Generated 2026-10-20T02:20:22.573Z