Changes / Honeycomb / Delayed ingestion of events in EU region
Honeycomb: Major incident: Delayed ingestion of events in EU region
Summary
Honeycomb reported major EU event-ingestion delays that could leave triggers, SLOs, and queries temporarily missing data. All events were processed after traffic rebalancing and scaling. The incident is resolved, but timestamps conflict: start 2025-08-13 17:10 UTC, resolution 2025-08-13 09:30 UTC; computed duration is unknown.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Delayed ingestion of events in EU region
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-08-13 17:10 UTC
Before: not present
After: 2025-08-13 09:30 UTC
Before: not present
After: For a period of approximately 1h30, from 9:34 UTC to 11:08 UTC, our ingestion of events in the EU region was delayed. All events were eventually processed, but for that duration, triggers, SLOs, and regular queries could have seen missing data for a brief period of time.
As we re-balanced traffic loads and scaled the system, ingestion caught up and data has since then been fully available. We are still actively working on improving the scaling margins and behaviors of the EU instance of Honeycomb to better sustain load variations we are observing at this time.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delayed ingestion of events in EU region (this change) Honeycomb reported major EU event-ingestion delays that could leave triggers, SLOs, and queries temporarily missing data. All events were processed after traffic rebalancing and scaling. The incident is resolved, but timestamps conflict: start 2025-08-13 17:10 UTC, resolution 2025-08-13 09:30 UTC; computed duration is unknown. | Major incident |
All Honeycomb changes → · Honeycomb profile · Markdown · JSON