Changes / Honeycomb / Ingest Outage in US and EU
Honeycomb: Major incident: Ingest Outage in US and EU
Summary
Honeycomb reported a major US and EU ingest outage starting 2026-06-24 18:14 UTC, resolved at 19:04 UTC after 50 min. Impact was primarily US: some telemetry was lost, triggers may not have fired, and SLO evaluations and Service Maps were delayed. Honeycomb rolled back a recent deploy; the postmortem reports telemetry gaps up to half an hour.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Ingest Outage in US and EU
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-06-24 18:14 UTC
Before: not present
After: 2026-06-24 19:04 UTC
Before: not present
After: api.eu1.honeycomb.io - EU1 Event Ingest, api.honeycomb.io - US1 Event Ingest, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting, ui.honeycomb.io - US1 Trigger & SLO Alerting
Before: not present
After: We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.
Before: not present
After: We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.
Before: not present
After: Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service
Before: not present
After: All services have been fully recovered.
Impact window: 17:55–18:17 UTC
Regions affected: Primary impact in US. The EU region was affected to a much lesser degree.
During this period, the following effects may have occurred:
Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window.
Triggers: Triggers may have failed to fire within the impact window.
SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn.
Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested.
We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.
Before: not present
After: On June 24, we experienced approximately 30 minutes of severe data ingestion degradation in Honeycomb’s US and EU instances. During this degradation, Honeycomb rejected a significant percentage of inbound telemetry, and customers would have seen up to a half-hour gap in ingested telemetry. Additionally, during this same time window, customers would have experienced data processing degradation and service disruption within Anomaly Detection and Service Maps.
A deployment containing a change to how services retrieve dataset schemas into local caches caused a cascading set of failures, resulting in failed remote cache retrievals, and subsequently, a MySQL stampede due to several services falling back to retrieving schemas from the database. This stampede resulted in significant memory utilization growth, causing some services to crash loop with out-of-memory exceptions while retrieving schemas. Customers would have seen 5xx errors from our ingestion APIs when this occurred, because ingestion services were among those that were crash looping. This also included crash looping from services responsible for Anomaly Detection and Service Maps.
Right after these service crash loops began, alerts fired for the crashing services, and procedures were run to pin all services back to the last known good build before the breaking change landed. Once services were running the previous deploy’s build, the 5xx error responses from ingestion APIs returned to nominal baseline levels, and services that were crash looping recovered and resumed normal operation.
The underlying issue centered on how dataset schema cache payloads and metadata were serialized to memcached for remote cache retrieval, and how memcached clients deserialize that data before writing it to a local cache. The code change contained a feature flag to toggle the serialization behavior, and backwards-compatibility safeguards for clients deserializing different schema versions from memcached. However, services’ schema deserialization prior to the feature flag flip on did not behave as intended, treating each backwards-compatible schema read as a hard error rather than as a fallback, which subsequently caused every remote schema cache read to fall through to the database. Subsequent analysis and review of the code identified the issue and we have re-deployed this change with a fix along with additional tests. Services now correctly and successfully deserialize schemas from memcached in a safe, backwards-compatible manner.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Ingest Outage in US and EU (this change) Honeycomb reported a major US and EU ingest outage starting 2026-06-24 18:14 UTC, resolved at 19:04 UTC after 50 min. Impact was primarily US: some telemetry was lost, triggers may not have fired, and SLO evaluations and Service Maps were delayed. Honeycomb rolled back a recent deploy; the postmortem reports telemetry gaps up to half an hour. | Major incident |
All Honeycomb changes → · Honeycomb profile · Markdown · JSON