Struck in red is what Lightstep (ServiceNow Cloud Observability) removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
AddedTitle#1
Before: not present
After: Failing to ingest some span data
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2022-07-21 10:15 UTC
AddedResolved time#5
Before: not present
After: 2022-07-21 11:41 UTC
AddedAffected components#6
Before: not present
After: Observability - Data Ingestion - Trace Assembly, US - Trace Assembly
AddedUpdateinvestigating#7
Before: not present
After: We have identified the issue and are monitoring a fix.
AddedUpdatemonitoring#8
Before: not present
After: A fix has been implemented and we are monitoring the results.
AddedUpdateresolved#9
Before: not present
After: This incident has been resolved.
AddedUpdatepostmortem#10
Before: not present
After: ## Summary
Increased span query load led to backpressure on span data writes. This backpressure led to resource exhaustion in one fleet of span ingestion pods, which caused a partial failure to ingest span data. The issue was resolved by scaling up the ingest pool.
## Timeline \(Pacific Time\)
_2:45AM: Expensive queries lead to a spike in CPU utilization for a database pool. This spike leads to load shedding for writes, causing span ingestion errors._
_3:14AM: Scaled up the CPU for the affected pool._
_3:33AM: Scaled up the span ingestion pods._
_4:15AM: All systems are healthy again._
## Action items
We have compiled a list of action items, including prioritizing ingest over queries and improving retry handling, to more gracefully handle increased load.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.