CompaniesLightstep (ServiceNow Cloud Observability)Incidents
Lightstep (ServiceNow Cloud Observability): Errors Ingesting Span Data
Affected components: Data Ingestion and Aggregation - StatisticsUS - Trace StatisticsData Ingestion and Aggregation - Trace AssemblyUS - Trace Assembly
Timeline of updates
- identified
The issue has been identified and a fix is being implemented.
- monitoring
A fix has been implemented and we are monitoring the results.
- resolved
This incident has been resolved.
- postmortem
## **Summary** A sharp increase in traffic overloaded one of our backend services responsible for communicating with satellites. The unavailability of the service led to a negative feedback loop as satellites attempted to reconnect to the failing service. During this time Lightstep experienced a partial outage of trace assembly and ingestion of statistics for streams. ## **Timeline \(12-hour Pacific Time\)** _~10:00a.m. Traffic to our SAAS sharply increases, overloading one of our backend services. 10:30a.m. The backend services restart, kicking off a cascading failure and starting the outage. 10:45a.m. Lightstep deploys more resources to an adjacent service which helps alleviate the symptoms of the outage and gets data flowing back into the system._ _11:20a.m. The rollout is complete and the cascading failure has resolved itself._ _12:07p.m. Lightstep marks the original incident as resolved as the symptoms have disappeared._ _12:24p.m. Lightstep increases resources for the root-cause service to reduce the likelihood of future outages. The change is deployed incorrectly and the service’s resources are decreased, which resumes the outage._ _12:28p.m. Lightstep updates the status page to reflect the new ongoing incident._ _12:51p.m. Lightstep identifies the error in the configuration, corrects it, and deploys the service with increased resources._ _12:58p.m. The service has finished rolling out and the outage is over._ **Action items** We have outlined 10 action items to avoid outages like this. Some examples include allocating more resources to the failing service, improving the autoscaling and alerting on the failing service, improving our autoscaling tools on adjacent services to be able to handle a failure in the root-cause service, and preventing OOMs of customer satellites when they cannot reach our SAAS. We’re also updating our incident response workflows to make it harder to incorrectly deploy bad configurations.
Changes Rewindex caught on this incident
- Feb 4, 2022Major incident: Errors Ingesting Span Data
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
This incident predates live tracking (October 17, 2026). It came from the vendor’s own published history, which is not covered by checkpoints.
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).
Lightstep (ServiceNow Cloud Observability) profileRecent Lightstep (ServiceNow Cloud Observability) incidentsMarkdownJSON