rewindex

ChangesHoneycombSlowness in US region

Honeycomb: Incident silently edited: Slowness in US region

Silent editNot classified yetChanged stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what Honeycomb removed; blue is what they added. Muted text is unchanged context.

ChangedUpdatepostmortem#1
Before: On October 20, Honeycomb experienced about 3 hours of degraded service in the US region.…
After: On October 20, we experienced roughly three hours of degraded service in the US region.…
Before: …the highest impact between 13:55 and 14:10 UTC. Some trigger alerts may have been late or missed.…
After: …the highest impact between 13:55 and 14:10 UTC. During this time, customers may have experienced slow or failed queries and a sluggish UI, and some trigger alerts may have been late or missed.…
Before: …some trigger alerts may have been late or missed during that time. SLO processing suffered delays between 13:26 and 14:10 UTC.  From 13:57 to 14:09 UTC, about half of…
After: …some trigger alerts may have been late or missed. From 13:26 to approximately 16:10 UTC, SLO processing was delayed, so SLO data may have appeared out of date and burn alerts may have been delayed or missed during that time. From 13:57 to 14:09 UTC, about half of…
Before: …requests to the event ingest API received 5xx errors. Lesser impact occurred between 13:48 to 13:55 \(1% error rate\) and for approximately minute at 14:15 \(3% error rate\). We’ll publish a more in-depth report on Friday, October 23 which will cover detection, remediation, and outstanding action items.
After: …requests to the event ingest API received 5xx errors, with lesser impact between 13:48 and 13:55 UTC (1% error rate) and for approximately a minute at 14:15 UTC (1.9% error rate). Telemetry rejected during these periods was not ingested unless the sender successfully retried it. customers may see gaps in their data for those periods, including in triggers and SLOs that cover it. Our on-call engineers were first paged at 13:43 UTC by an alert about triggers not running on time, and we spun up our internal incident response as further alerts followed. Our investigation found that the primary database’s CPU was saturated by a large number of long-running read queries. We performed a database failover to relieve the load. This briefly eased the pressure, but the same queries soon reappeared on the new primary, so it did not resolve the problem. We then reduced database load by terminating those queries and temporarily pausing SLO processing, which restored the other affected services. Once a database-side timeout was in place on the responsible query, we resumed SLO processing, which then caught up, and we repaired historical SLO data. These queries came from a check our SLO processing runs whenever a Honeycomb environment's setti …

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdayFriday, October 23Fri, Oct 23

1 rewrite

Wednesday, October 21

1 incident change

Tuesday, October 20

3 incident changes

All Honeycomb changesHoneycomb profileMarkdownJSON