Lightstep (ServiceNow Cloud Observability): Failing to ingest some span data

MajorStatus: postmortemStarted 2022-07-21 10:15 UTCResolved 2022-07-21 11:41 UTC (1.4 h)4 updatesstspg.io

Affected components: Observability - Data Ingestion - Trace AssemblyUS - Trace Assembly

Timeline of updates

investigating2022-07-21 10:15 UTC
We have identified the issue and are monitoring a fix.
monitoring2022-07-21 10:49 UTC
A fix has been implemented and we are monitoring the results.
resolved2022-07-21 11:41 UTC
This incident has been resolved.
postmortem2022-07-29 18:36 UTC
## Summary Increased span query load led to backpressure on span data writes. This backpressure led to resource exhaustion in one fleet of span ingestion pods, which caused a partial failure to ingest span data. The issue was resolved by scaling up the ingest pool. ## Timeline \(Pacific Time\) _2:45AM: Expensive queries lead to a spike in CPU utilization for a database pool. This spike leads to load shedding for writes, causing span ingestion errors._ _3:14AM: Scaled up the CPU for the affected pool._ _3:33AM: Scaled up the span ingestion pods._ _4:15AM: All systems are healthy again._ ## Action items We have compiled a list of action items, including prioritizing ingest over queries and improving retry handling, to more gracefully handle increased load.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).