Honeycomb: Honeycomb Ingest Outage
AWS outage, Jul 28, 2020 · 18 vendors, 18 incidents
Honeycomb’s incident blames AWS as the cause. (hop 1, posted +0 min vs the first vendor naming AWS)
“…ours and impacting 0.002% of ingest traffic over that period. The outage was triggered by AWS Route53 suffering lookup failures from EC2 instances in the us-east-1a availability zone (AZ), which hosts…”
Swipe sideways to see the whole diagram.
Mermaid source
flowchart LR
root["AWS<br/>own incident not archived"]:::inferred
h1["Within an hour<br/>3 vendors, 3 blamed AWS<br/>Honeycomb, Twilio, Chargebee"]:::blames
root --> h1
h4["1 to 4 hours later<br/>2 vendors, 1 blamed AWS<br/>Stream (GetStream), Elastic Cloud"]:::blames
root --> h4
me["Honeycomb (this incident)<br/>posted +0 min"]:::me
h1 --- meFull event: timeline, evidence and all vendors →
Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.
Timeline of updates
- resolved
On July 28 starting at approximately 01:30am PDT, Honeycomb began experiencing increased error rates and latency across our services, leading to a partial outage ultimately lasting approximately two hours and impacting 0.002% of ingest traffic over that period. The outage was triggered by AWS Route53 suffering lookup failures from EC2 instances in the us-east-1a availability zone (AZ), which hosts many of our instances across all our services. Our Ingest Latency SLO burn alert triggered at 01:53am PDT. On-call investigated and used Honeycomb (BubbleUp specifically) to determine that our ingest instances in us-east-1a were to blame. On-call responded by removing that AZ from the autoscaling group, which reacted by replacing the problematic hosts with new ones in unaffected AZs. This migration finished at 03:15am PDT. There was a small amount of ingested event loss due to the outage (roughly 35000 event payloads, 0.002% of our traffic over that period) because retries for sending the events to Kafka were exhausted on the affected ingest hosts. While ingest is the most important service to fix quickly, this outage also affected other services: We saw elevated response times for our UI and storage service. Some triggers and burn alerts also failed to evaluate during that time window. If you believe you have been impacted by a trigger or alert failing to run during this period, please reach out to our support team at honeycomb.io/support. Throughout this incident, while degraded, Honeycomb remained usable and we used it to debug the incident. AWS declared the outage resolved at 04:37am PDT, but we delayed transitioning ingest hosts back to us-east-1a until 09:45am PDT.
From Honeycomb’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.
Impact, status and times are the vendor’s own, as last archived. View on status.honeycomb.io (opens in new tab).