# Lightstep (ServiceNow Cloud Observability): Critical incident: Lightstep Webapp Down

**Summary:** _summary pending_
- Kind: Major incident · Label: critical (vendor)
- Changed: 2022-01-20 17:58 UTC · Detected: 2026-10-19 18:33 UTC
- [Source: https://stspg.io/0ywk8cvcqg82](https://stspg.io/0ywk8cvcqg82)
- [Company: Lightstep (ServiceNow Cloud Observability)](https://rewindex.com/company/lightstep.md)
- Versions: before (none: first archived) → after incident\_version:742/mvbs7hjyddbt@ffa589b92ed2
- Change 1 of 1 for this incident

Incident page (full timeline): [Lightstep Webapp Down](https://rewindex.com/company/lightstep/incidents/mvbs7hjyddbt.md)
## What changed

**1. Added: Title**

```diff
+ Lightstep Webapp Down
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2022-01-20 17:58 UTC
```

**5. Added: Resolved time**

```diff
+ 2022-01-20 21:21 UTC
```

**6. Added: Affected components**

```diff
+ US - Change Intelligence, US - Dashboards, US - Explorer, US - Service Directory, Web UI - Dashboards, Web UI - Explorer, Web UI - Login, Web UI - Root Cause Analysis, Web UI - Service Directory
```

**7. Added: Update** (investigating) · 2022-01-20 17:58 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (investigating) · 2022-01-20 18:41 UTC

```diff
+ All pages are operational except for the streams page.  We are continuing to investigate.
```

**9. Added: Update** (monitoring) · 2022-01-20 20:08 UTC

```diff
+ A fix for the remaining unavailability on the streams page is rolling out, and is expected to complete by 1pm PST.
```

**10. Added: Update** (monitoring) · 2022-01-20 21:21 UTC

```diff
+ We are continuing to monitor for any further issues.
```

**11. Added: Update** (resolved) · 2022-01-20 21:21 UTC

```diff
+ This incident has been resolved.
```

**12. Added: Update** (postmortem) · 2022-02-01 23:14 UTC

```diff
+ ## **Summary** Web UI inaccessible for 48 minutes \(9:49am - 10:37am\), and streams page inaccessible for an additional 48 minutes \(until 11:25am\). ## **Timeline \(12-hour Pacific Time\)** 09:49 AM PT: 100% of API requests for the webapp begin failing 09:52 AM PT: Database hits 100% resource utilization 10:31 AM PT: Divert all traffic from webapp to bring up database 10:37 AM PT: Allow all traffic except from the operation/get endpoint so the webapp is back up 11:25 AM PT: Allow operation/get traffic and incident is resolved ## **Root Cause** Database errors led to transient unavailability.  Aggressive retries saturated the database, leading to a negative feedback loop. ## **Action items** We have updated our database configuration to limit the blast radius of failing or slow database calls.  We have also implemented additional rate limiting to avoid the negative feedback loop where many concurrent requests lead to failures, leading to more requests.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2022-01-20 17:58 UTC | Critical incident: Lightstep Webapp Down (this change) | critical |

All Lightstep (ServiceNow Cloud Observability) changes: [timeline](https://rewindex.com/company/lightstep/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i386d4141be2d5d · Markdown: https://rewindex.com/change/i386d4141be2d5d.md · JSON: https://rewindex.com/api/v1/changes/i386d4141be2d5d · Generated 2026-10-20T00:48:39.248Z