Companies/Honeycomb/Incidents

Honeycomb: Slowness in US region

Affected components: ui.honeycomb.io - US1 Queryingui.honeycomb.io - US1 Trigger & SLO Alertingui.honeycomb.io - US1 App Interface

Timeline of updates

1. investigating2026-10-20 14:04 UTC

   We are currently investigating the degradation of various services in our US region.

2. investigating2026-10-20 15:18 UTC

   We've identified and isolated the impacted service and are seeing recovery in query execution. SLO processing remains degraded.

3. resolved2026-10-20 16:26 UTC

   SLO processing has fully recovered. All services are now functional.

4. postmortem2026-10-21 23:26 UTCEdited

   On October 20, we experienced roughly three hours of degraded service in the US region.
   From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. During this time, customers may have experienced slow or failed queries and a sluggish UI, and some trigger alerts may have been late or missed.
   From 13:26 to approximately 16:10 UTC, SLO processing was delayed, so SLO data may have appeared out of date and burn alerts may have been delayed or missed during that time.
   From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors, with lesser impact between 13:48 and 13:55 UTC (1% error rate) and for approximately a minute at 14:15 UTC (1.9% error rate).
   Telemetry rejected during these periods was not ingested unless the sender successfully retried it. customers may see gaps in their data for those periods, including in triggers and SLOs that cover it.
   Our on-call engineers were first paged at 13:43 UTC by an alert about triggers not running on time, and we spun up our internal incident response as further alerts followed.
   Our investigation found that the primary database’s CPU was saturated by a large number of long-running read queries. We performed a database failover to relieve the load. This briefly eased the pressure, but the same queries soon reappeared on the new primary, so it did not resolve the problem. We then reduced database load by terminating those queries and temporarily pausing SLO processing, which restored the other affected services. Once a database-side timeout was in place on the responsible query, we resumed SLO processing, which then caught up, and we repaired historical SLO data.
   These queries came from a check our SLO processing runs whenever a Honeycomb environment's settings change. In a vast majority of cases, it completes in a few milliseconds. For the 0.1% of our environments with an unusually large number of SLOs and calculated fields, however, the database ran it in a very inefficient way, and a routine update to one such environment's settings made the check too slow to ever complete. These queries continued running after the requesting service timed out, and repeated attempts increased the load on a database shared by several services.
   To prevent this from happening again, in addition to the database-side timeout, we changed how the database executes the query for environments like this one, which brought its runtime down from minutes to a few milliseconds. We have also adjusted our alerting thresholds to ensure that an on-call engineer is made aware of future database load changes quickly.

Changes Rewindex caught on this incident

• Oct 20Major incident: Slowness in US region
• Oct 20Incident updated: Slowness in US region
• Oct 20Incident updated: Slowness in US region
• Oct 21Incident updated: Slowness in US region
• Oct 23Incident silently edited: Slowness in US region

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Honeycomb profileRecent Honeycomb incidentsMarkdownJSON