Struck in red is what Lightstep (ServiceNow Cloud Observability) removed; blue is what they added. Muted text is unchanged context.
BeforeAfter
AddedTitle#1
Before: not present
After: Ingestion and web UI outage
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2022-07-21 20:05 UTC
AddedResolved time#5
Before: not present
After: 2022-07-21 21:02 UTC
AddedAffected components#6
Before: not present
After: Observability - Alerting - Metric alerting, Observability - Alerting - Trace alerting, Observability - API, Observability - Data Ingestion - Metrics, Observability - Data Ingestion - Trace Assembly, Observability - Data Ingestion - Trace Statistics, Observability - Third-Party Notification Methods - PagerDuty Events API, Observability - Third-Party Notification Methods - Slack Apps/Integrations, Observability - Third-Party Notification Methods - Slack Connections, Observability - Third-Party Notification Methods - Slack Messaging, Observability - Web UI - Change Intelligence, Observability - Web UI - Dashboards, Observability - Web UI - Explorer, Observability - Web UI - Notebooks, Observability - Web UI - Service Directory, PagerDuty Events API, Slack Apps/Integrations, Slack Connections, Slack Messaging, US - Change Intelligence, US - Dashboards, US - Explorer, US - Metrics, US - Notebooks, US - Observability - API, US - Service Directory, US - Streams Alerting, US - Trace Assembly, US - Trace Statistics, US - Unified Alerting
AddedUpdateinvestigating#7
Before: not present
After: We are currently investigating this issue.
AddedUpdateinvestigating#8
Before: not present
After: We are continuing to investigate this issue.
AddedUpdateinvestigating#9
Before: not present
After: We are still actively investigating this issue, and will provide another update shortly.
AddedUpdateinvestigating#10
Before: not present
After: We have now identified the issue and are actively implementing mitigations.
AddedUpdateidentified#11
Before: not present
After: We have mitigated issues with metric ingestion. Trace ingestion, assembly and alerting are still experiencing degraded performance for some customers.
AddedUpdatemonitoring#12
Before: not present
After: We have resolved issues with trace ingestion, assembly and alerting and believe the issue has been resolved, but are actively monitoring.
AddedUpdateresolved#13
Before: not present
After: All components are operational.
AddedUpdatepostmortem#14
Before: not present
After: ### Summary
Lightstep UI and Data Ingest experienced an outage triggered by a planned database change on July 21st. This database change caused a service failure, which led to a cascading failure in dependent services. Action was taken to restore the service in a degraded mode, which recovered the Lightstep UI. A subsequent roll back of the database change restored all systems and services.
### Timeline
12:22 PM: A database change was run causing a service to become unavailable.
12:25 PM: Cascading failures impact Lightstep UI and ingestion. Ingested data loss begins.
12:35 PM: Incident declared in status page.
12:49 PM: Root cause identified, service brought back in a degraded mode, recovery begins.
13:01 PM: Several systems stabilize, Lightstep UI recovers.
13:17 PM: Database change rolled back. Remaining affected services begin recovering.
13:30 PM: Ingestion recovers. Incident resolved, status page updated to “mitigated and monitoring”.
14:02 PM: Status page updated to fully operational.
### Action Items
* Adding automated checks on code changes with database changes to help prevent the original root cause
* Enabling the directly affected service to continue to serve traffic in a degraded state, rather than failing entirely.
* Making architectural changes to Lightstep’s Data Ingest path to prevent data loss in these situations by continuing to accept and buffer incoming traffic.
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.