Struck in red is what Lightstep (ServiceNow Cloud Observability) removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
AddedTitle#1
Before: not present
After: Lightstep Webapp Down
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2022-01-20 17:58 UTC
AddedResolved time#5
Before: not present
After: 2022-01-20 21:21 UTC
AddedAffected components#6
Before: not present
After: US - Change Intelligence, US - Dashboards, US - Explorer, US - Service Directory, Web UI - Dashboards, Web UI - Explorer, Web UI - Login, Web UI - Root Cause Analysis, Web UI - Service Directory
AddedUpdateinvestigating#7
Before: not present
After: We are currently investigating this issue.
AddedUpdateinvestigating#8
Before: not present
After: All pages are operational except for the streams page. We are continuing to investigate.
AddedUpdatemonitoring#9
Before: not present
After: A fix for the remaining unavailability on the streams page is rolling out, and is expected to complete by 1pm PST.
AddedUpdatemonitoring#10
Before: not present
After: We are continuing to monitor for any further issues.
AddedUpdateresolved#11
Before: not present
After: This incident has been resolved.
AddedUpdatepostmortem#12
Before: not present
After: ## **Summary**
Web UI inaccessible for 48 minutes \(9:49am - 10:37am\), and streams page inaccessible for an additional 48 minutes \(until 11:25am\).
## **Timeline \(12-hour Pacific Time\)**
09:49 AM PT: 100% of API requests for the webapp begin failing
09:52 AM PT: Database hits 100% resource utilization
10:31 AM PT: Divert all traffic from webapp to bring up database
10:37 AM PT: Allow all traffic except from the operation/get endpoint so the webapp is back up
11:25 AM PT: Allow operation/get traffic and incident is resolved
## **Root Cause**
Database errors led to transient unavailability. Aggressive retries saturated the database, leading to a negative feedback loop.
## **Action items**
We have updated our database configuration to limit the blast radius of failing or slow database calls. We have also implemented additional rate limiting to avoid the negative feedback loop where many concurrent requests lead to failures, leading to more requests.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.