# Lightstep (ServiceNow Cloud Observability): Major incident: Ingestion and web UI outage

**Summary:** _summary pending_
- Kind: Major incident · Label: major (vendor)
- Changed: 2022-07-21 20:05 UTC · Detected: 2026-10-19 18:33 UTC
- [Source: https://stspg.io/zdcb2q1tz85w](https://stspg.io/zdcb2q1tz85w)
- [Company: Lightstep (ServiceNow Cloud Observability)](https://rewindex.com/company/lightstep.md)
- Versions: before (none: first archived) → after incident\_version:742/jyj858ftvg2p@5a42dcc690b3
- Change 1 of 1 for this incident

Incident page (full timeline): [Ingestion and web UI outage](https://rewindex.com/company/lightstep/incidents/jyj858ftvg2p.md)
## What changed

**1. Added: Title**

```diff
+ Ingestion and web UI outage
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2022-07-21 20:05 UTC
```

**5. Added: Resolved time**

```diff
+ 2022-07-21 21:02 UTC
```

**6. Added: Affected components**

```diff
+ Observability - Alerting - Metric alerting, Observability - Alerting - Trace alerting, Observability - API, Observability - Data Ingestion - Metrics, Observability - Data Ingestion - Trace Assembly, Observability - Data Ingestion - Trace Statistics, Observability - Third-Party Notification Methods - PagerDuty Events API, Observability - Third-Party Notification Methods - Slack Apps/Integrations, Observability - Third-Party Notification Methods - Slack Connections, Observability - Third-Party Notification Methods - Slack Messaging, Observability - Web UI - Change Intelligence, Observability - Web UI - Dashboards, Observability - Web UI - Explorer, Observability - Web UI - Notebooks, Observability - Web UI - Service Directory, PagerDuty Events API, Slack Apps/Integrations, Slack Connections, Slack Messaging, US - Change Intelligence, US - Dashboards, US - Explorer, US - Metrics, US - Notebooks, US - Observability - API, US - Service Directory, US - Streams Alerting, US - Trace Assembly, US - Trace Statistics, US - Unified Alerting
```

**7. Added: Update** (investigating) · 2022-07-21 19:35 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (investigating) · 2022-07-21 19:35 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (investigating) · 2022-07-21 19:48 UTC

```diff
+ We are still actively investigating this issue, and will provide another update shortly.
```

**10. Added: Update** (investigating) · 2022-07-21 20:05 UTC

```diff
+ We have now identified the issue and are actively implementing mitigations.
```

**11. Added: Update** (identified) · 2022-07-21 20:20 UTC

```diff
+ We have mitigated issues with metric ingestion.  Trace ingestion, assembly and alerting are still experiencing degraded performance for some customers.
```

**12. Added: Update** (monitoring) · 2022-07-21 20:30 UTC

```diff
+ We have resolved issues with trace ingestion, assembly and alerting and believe the issue has been resolved, but are actively monitoring.
```

**13. Added: Update** (resolved) · 2022-07-21 21:02 UTC

```diff
+ All components are operational.
```

**14. Added: Update** (postmortem) · 2022-07-29 00:21 UTC

```diff
+ ### Summary Lightstep UI and Data Ingest experienced an outage triggered by a planned database change on July 21st. This database change caused a service failure, which led to a cascading failure in dependent services. Action was taken to restore the service in a degraded mode, which recovered the Lightstep UI. A subsequent roll back of the database change restored all systems and services. ### Timeline 12:22 PM: A database change was run causing a service to become unavailable. 12:25 PM: Cascading failures impact Lightstep UI and ingestion. Ingested data loss begins. 12:35 PM: Incident declared in status page. 12:49 PM: Root cause identified, service brought back in a degraded mode, recovery begins. 13:01 PM: Several systems stabilize, Lightstep UI recovers. 13:17 PM: Database change rolled back. Remaining affected services begin recovering. 13:30 PM: Ingestion recovers. Incident resolved, status page updated to “mitigated and monitoring”. 14:02 PM: Status page updated to fully operational. ### Action Items * Adding automated checks on code changes with database changes to help prevent the original root cause * Enabling the directly affected service to continue to serve traffic in a degraded state, rather than failing entirely. * Making architectural changes to Lightstep’s Data Ingest path to prevent data loss in these situations by continuing to accept and buffer incoming traffic.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2022-07-21 20:05 UTC | Major incident: Ingestion and web UI outage (this change) | major |

All Lightstep (ServiceNow Cloud Observability) changes: [timeline](https://rewindex.com/company/lightstep/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ia8f10b64aef95c · Markdown: https://rewindex.com/change/ia8f10b64aef95c.md · JSON: https://rewindex.com/api/v1/changes/ia8f10b64aef95c · Generated 2026-10-20T00:48:47.189Z