Changes / Honeycomb / Delays in SLO, Service Maps processing
Honeycomb: Major incident: Delays in SLO, Service Maps processing
Summary
Honeycomb reported a major US1 degradation starting 2025-10-20 14:16 UTC, with SLO and Service Maps processing delays exceeding 5 minutes and later querying and trigger delays. Constrained EC2 capacity in AWS us-east-1 was cited; resolution came at 2025-10-21 15:41 UTC after 25 h 24 min, with no customer data reaching its load balancers lost.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 89%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Delays in SLO, Service Maps processing
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-10-20 14:16 UTC
Before: not present
After: 2025-10-21 15:41 UTC
Before: not present
After: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Trigger & SLO Alerting
Before: not present
After: Due to constrained EC2 instance capacity in the AWS us-east-1 region, we are choosing to allocate the capacity we have to incoming events and telemetry. As such, customers may see over 5 minute delays in our processing of
- SLOs
- Service Maps
We do not expect a degradation of our core ingest -> query flow, and we do not expect triggers to be impacted
Before: not present
After: SLO processing has recovered. Service Maps continues to be degraded but historical data can be queried.
Before: not present
After: The networking issues in us-east-1 is affecting our query engine - customers may see errors or delays when running queries as a result.
Before: not present
After: We’re currently observing that querying is seeing signs of recovery. We are continuing to monitor the situation
Before: not present
After: AWS is starting to show signs of recovery. Querying remains partially impacted and may take longer to return results. We are continuing to monitor the situation.
Before: not present
After: Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. We are continuing to monitor the situation.
Before: not present
After: Honeycomb core functionality is operational. Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. This is our final update for the night unless the situation degrades. We will continue to monitor the situation.
Before: not present
After: This incident has been resolved. At no point did we lose any customer data that hit our load balancers. Querying has been stable for hours, and all features that were degraded are functional.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delays in SLO, Service Maps processing (this change) Honeycomb reported a major US1 degradation starting 2025-10-20 14:16 UTC, with SLO and Service Maps processing delays exceeding 5 minutes and later querying and trigger delays. Constrained EC2 capacity in AWS us-east-1 was cited; resolution came at 2025-10-21 15:41 UTC after 25 h 24 min, with no customer data reaching its load balancers lost. | Major incident |
All Honeycomb changes → · Honeycomb profile · Markdown · JSON