# Honeycomb: Honeycomb Ingest Outage

- Impact (vendor’s label): minor
- Status: resolved
- Started: 2020-07-28 08:30 UTC · Resolved: 2020-07-28 08:30 UTC · Duration: 0 min
- Updates: 1
- [Vendor status page: https://status.honeycomb.io/incidents/jls2lggldlfj](https://status.honeycomb.io/incidents/jls2lggldlfj)
- [Company: Honeycomb](https://rewindex.com/company/honeycomb.md)

## Part of an outage event

- [AWS outage, Jul 28, 2020: 18 vendors, 18 incidents](https://rewindex.com/events/2020-07-28-aws)
- This incident blames AWS as the cause. (hop 1, posted +0 min vs the first vendor naming AWS)
- Evidence: “…ours and impacting 0.002% of ingest traffic over that period. The outage was triggered by AWS Route53 suffering lookup failures from EC2 instances in the us-east-1a availability zone (AZ), which hosts…”

```mermaid
flowchart LR
  root["AWS<br/>own incident not archived"]:::inferred
  h1["Within an hour<br/>3 vendors, 3 blamed AWS<br/>Honeycomb, Twilio, Chargebee"]:::blames
  root --> h1
  h4["1 to 4 hours later<br/>2 vendors, 1 blamed AWS<br/>Stream (GetStream), Elastic Cloud"]:::blames
  root --> h4
  me["Honeycomb (this incident)<br/>posted +0 min"]:::me
  h1 --- me
```

_Detected automatically from incidents that started together and named the same provider; a link means one incident named another vendor, not a confirmed cause._

## Timeline of updates

- **resolved** 2020-07-28 08:30 UTC: On July 28 starting at approximately 01:30am PDT, Honeycomb began experiencing increased error rates and latency across our services, leading to a partial outage ultimately lasting approximately two hours and impacting 0.002% of ingest traffic over that period. The outage was triggered by AWS Route53 suffering lookup failures from EC2 instances in the us-east-1a availability zone (AZ), which hosts many of our instances across all our services. Our Ingest Latency SLO burn alert triggered at 01:53am PDT. On-call investigated and used Honeycomb (BubbleUp specifically) to determine that our ingest instances in us-east-1a were to blame. On-call responded by removing that AZ from the autoscaling group, which reacted by replacing the problematic hosts with new ones in unaffected AZs. This migration finished at 03:15am PDT. There was a small amount of ingested event loss due to the outage (roughly 35000 event payloads, 0.002% of our traffic over that period) because retries for sending the events to Kafka were exhausted on the affected ingest hosts. While ingest is the most important service to fix quickly, this outage also affected other services: We saw elevated response times for our UI and storage service. Some triggers and burn alerts also failed to evaluate during that time window. If you believe you have been impacted by a trigger or alert failing to run during this period, please reach out to our support team at honeycomb.io/support. Throughout this incident, while degraded, Honeycomb remained usable and we used it to debug the incident. AWS declared the outage resolved at 04:37am PDT, but we delayed transitioning ingest hosts back to us-east-1a until 09:45am PDT.

_From the vendor’s published history: only the final update text is available for incidents before Rewindex began polling this page._


---
Canonical: https://rewindex.com/company/honeycomb/incidents/jls2lggldlfj · Markdown: https://rewindex.com/company/honeycomb/incidents/jls2lggldlfj.md · JSON: https://rewindex.com/api/v1/companies/honeycomb/incidents/jls2lggldlfj · Generated 2026-10-20T00:09:57.524Z