rewindex

ChangesHoneycombSlowness in US region

Honeycomb: Incident updated: Slowness in US region

Incident updateNormal progress (rule, 100%)Changed stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

What changed

Struck in red is what Honeycomb removed; blue is what they added. Muted text is unchanged context.

ChangedStatus#1
Before: resolved
After: postmortem
AddedUpdatepostmortem#2
Before: not present
After: On October 20, Honeycomb experienced about 3 hours of degraded service in the US region. From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. Some trigger alerts may have been late or missed during that time. SLO processing suffered delays between 13:26 and 14:10 UTC.  From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors. Lesser impact occurred between 13:48 to 13:55 \(1% error rate\) and for approximately minute at 14:15 \(3% error rate\). We’ll publish a more in-depth report on Friday, October 23 which will cover detection, remediation, and outstanding action items.

Compared with the previous archived version of this incident (10 to 30-minute polling). View the live incident (opens in new tab).

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves these records existed by then and have not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

History of this incident

YesterdayFriday, October 23Fri, Oct 23

1 rewrite

Wednesday, October 21

1 incident change

Tuesday, October 20

3 incident changes

All Honeycomb changesHoneycomb profileMarkdownJSON