Companies / Honeycomb / incidents
Honeycomb: Ingest Outage in US and EU
Honeycomb reported a major US and EU ingest outage starting 2026-06-24 18:14 UTC, resolved at 19:04 UTC after 50 min. Impact was primarily US: some telemetry was lost, triggers may not have fired, and SLO evaluations and Service Maps were delayed. Honeycomb rolled back a recent deploy; the postmortem reports telemetry gaps up to half an hour.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: api.honeycomb.io - US1 Event Ingestui.honeycomb.io - US1 Trigger & SLO Alertingapi.eu1.honeycomb.io - EU1 Event Ingestui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting
Timeline of updates
- investigating
We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.
- monitoring
We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.
- monitoring
Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service
- resolved
All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.
- postmortem
On June 24, we experienced approximately 30 minutes of severe data ingestion degradation in Honeycomb’s US and EU instances. During this degradation, Honeycomb rejected a significant percentage of inbound telemetry, and customers would have seen up to a half-hour gap in ingested telemetry. Additionally, during this same time window, customers would have experienced data processing degradation and service disruption within Anomaly Detection and Service Maps. A deployment containing a change to how services retrieve dataset schemas into local caches caused a cascading set of failures, resulting in failed remote cache retrievals, and subsequently, a MySQL stampede due to several services falling back to retrieving schemas from the database. This stampede resulted in significant memory utilization growth, causing some services to crash loop with out-of-memory exceptions while retrieving schemas. Customers would have seen 5xx errors from our ingestion APIs when this occurred, because ingestion services were among those that were crash looping. This also included crash looping from services responsible for Anomaly Detection and Service Maps. Right after these service crash loops began, alerts fired for the crashing services, and procedures were run to pin all services back to the last known good build before the breaking change landed. Once services were running the previous deploy’s build, the 5xx error responses from ingestion APIs returned to nominal baseline levels, and services that were crash looping recovered and resumed normal operation. The underlying issue centered on how dataset schema cache payloads and metadata were serialized to memcached for remote cache retrieval, and how memcached clients deserialize that data before writing it to a local cache. The code change contained a feature flag to toggle the serialization behavior, and backwards-compatibility safeguards for clients deserializing different schema versions from memcached. However, services’ schema deserialization prior to the feature flag flip on did not behave as intended, treating each backwards-compatible schema read as a hard error rather than as a fallback, which subsequently caused every remote schema cache read to fall through to the database. Subsequent analysis and review of the code identified the issue and we have re-deployed this change with a fix along with additional tests. Services now correctly and successfully deserialize schemas from memcached in a safe, backwards-compatible manner.
Rewindex changes for this incident
- Major incident: Ingest Outage in US and EU 2026-06-24 18:14 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Honeycomb profile · Recent Honeycomb incidents · Markdown · JSON