Changes / Honeycomb / Ingest issues for some customers
Honeycomb: Major incident: Ingest issues for some customers
Summary
Honeycomb reported a major US1 Event Ingest outage affecting data ingestion for some teams on Classic environments, from 2026-04-10 13:41 UTC to 2026-04-10 17:24 UTC (3 h 42 min). Honeycomb applied a manual fix, then confirmed the fix covered all impacted and potentially impacted environments and related ingest errors should no longer occur.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Ingest issues for some customers
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-04-10 13:41 UTC
Before: not present
After: 2026-04-10 17:24 UTC
Before: not present
After: api.honeycomb.io - US1 Event Ingest
Before: not present
After: We have confirmed reports of some teams getting issues ingesting their data. We have identified a probable source behind this behavior and are currently trying to correct it.
Before: not present
After: We have further narrowed down that only teams on Classic environments are currently seeing issues. We are figuring out mitigation mechanisms at the moment.
Before: not present
After: We've identified a manual fix and are deploying that by hand to each affected environment, while also investigating a systemic solution.
Before: not present
After: We've applied the manual fix to nearly all affected datasets (less than 1% remaining). The systemic solution is still in progress.
Before: not present
After: The manual fix has been applied to all environments.
Before: not present
After: We are continuing to work on the systemic solution and monitor. We'll leave the incident open until we're able to confirm the issue won't recur.
Before: not present
After: We can confirm that the fix was applied uniformly to all impacted and potentially impacted environments; no teams should see ingest errors related to this issue anymore.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Ingest issues for some customers (this change) Honeycomb reported a major US1 Event Ingest outage affecting data ingestion for some teams on Classic environments, from 2026-04-10 13:41 UTC to 2026-04-10 17:24 UTC (3 h 42 min). Honeycomb applied a manual fix, then confirmed the fix covered all impacted and potentially impacted environments and related ingest errors should no longer occur. | Major incident |
All Honeycomb changes → · Honeycomb profile · Markdown · JSON