# Lightstep (ServiceNow Cloud Observability): Failing to ingest some span data

- Impact (vendor’s label): major
- Status: postmortem
- Started: 2022-07-21 10:15 UTC · Resolved: 2022-07-21 11:41 UTC · Duration: 1.4 h
- Updates: 4 · Affected components: Observability - Data Ingestion - Trace Assembly, US - Trace Assembly
- [Vendor status page: https://stspg.io/66899fzvzhw7](https://stspg.io/66899fzvzhw7)
- [Company: Lightstep (ServiceNow Cloud Observability)](https://rewindex.com/company/lightstep.md)

## Timeline of updates

- **investigating** 2022-07-21 10:15 UTC: We have identified the issue and are monitoring a fix.
- **monitoring** 2022-07-21 10:49 UTC: A fix has been implemented and we are monitoring the results.
- **resolved** 2022-07-21 11:41 UTC: This incident has been resolved.
- **postmortem** 2022-07-29 18:36 UTC: \#\# Summary Increased span query load led to backpressure on span data writes. This backpressure led to resource exhaustion in one fleet of span ingestion pods, which caused a partial failure to ingest span data. The issue was resolved by scaling up the ingest pool. \#\# Timeline \\(Pacific Time\\) \_2:45AM: Expensive queries lead to a spike in CPU utilization for a database pool.  This spike leads to load shedding for writes, causing span ingestion errors.\_ \_3:14AM: Scaled up the CPU for the affected pool.\_ \_3:33AM: Scaled up the span ingestion pods.\_ \_4:15AM: All systems are healthy again.\_ \#\# Action items We have compiled a list of action items, including prioritizing ingest over queries and improving retry handling, to more gracefully handle increased load.

## Rewindex changes for this incident

- 2022-07-21 10:15 UTC: [Major incident: Failing to ingest some span data](https://rewindex.com/change/i89d8cdc253c109.md)


---
Canonical: https://rewindex.com/company/lightstep/incidents/jyk6rhl7sq68 · Markdown: https://rewindex.com/company/lightstep/incidents/jyk6rhl7sq68.md · JSON: https://rewindex.com/api/v1/companies/lightstep/incidents/jyk6rhl7sq68 · Generated 2026-10-20T00:03:36.172Z