# Lightstep (ServiceNow Cloud Observability): Major incident: Failing to ingest some span data

**Summary:** _summary pending_
- Kind: Major incident · Label: major (vendor)
- Changed: 2022-07-21 10:15 UTC · Detected: 2026-10-19 18:33 UTC
- [Source: https://stspg.io/66899fzvzhw7](https://stspg.io/66899fzvzhw7)
- [Company: Lightstep (ServiceNow Cloud Observability)](https://rewindex.com/company/lightstep.md)
- Versions: before (none: first archived) → after incident\_version:742/jyk6rhl7sq68@f858a53ab1ed
- Change 1 of 1 for this incident

Incident page (full timeline): [Failing to ingest some span data](https://rewindex.com/company/lightstep/incidents/jyk6rhl7sq68.md)
## What changed

**1. Added: Title**

```diff
+ Failing to ingest some span data
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2022-07-21 10:15 UTC
```

**5. Added: Resolved time**

```diff
+ 2022-07-21 11:41 UTC
```

**6. Added: Affected components**

```diff
+ Observability - Data Ingestion - Trace Assembly, US - Trace Assembly
```

**7. Added: Update** (investigating) · 2022-07-21 10:15 UTC

```diff
+ We have identified the issue and are monitoring a fix.
```

**8. Added: Update** (monitoring) · 2022-07-21 10:49 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**9. Added: Update** (resolved) · 2022-07-21 11:41 UTC

```diff
+ This incident has been resolved.
```

**10. Added: Update** (postmortem) · 2022-07-29 18:36 UTC

```diff
+ ## Summary Increased span query load led to backpressure on span data writes. This backpressure led to resource exhaustion in one fleet of span ingestion pods, which caused a partial failure to ingest span data. The issue was resolved by scaling up the ingest pool. ## Timeline \(Pacific Time\) _2:45AM: Expensive queries lead to a spike in CPU utilization for a database pool.  This spike leads to load shedding for writes, causing span ingestion errors._ _3:14AM: Scaled up the CPU for the affected pool._ _3:33AM: Scaled up the span ingestion pods._ _4:15AM: All systems are healthy again._ ## Action items We have compiled a list of action items, including prioritizing ingest over queries and improving retry handling, to more gracefully handle increased load.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2022-07-21 10:15 UTC | Major incident: Failing to ingest some span data (this change) | major |

All Lightstep (ServiceNow Cloud Observability) changes: [timeline](https://rewindex.com/company/lightstep/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i89d8cdc253c109 · Markdown: https://rewindex.com/change/i89d8cdc253c109.md · JSON: https://rewindex.com/api/v1/changes/i89d8cdc253c109 · Generated 2026-10-20T00:48:39.430Z