Companies / Honeycomb / incidents
Honeycomb: Querying Errors
Timeline of updates
- resolved
On September 28, we received alerts that two of our query instance hosts on one of our partitions had died. Since they died close after each other, we initially suspected a problem with some data that needed processing, and tried restarting the process - which failed - and then tried replacing one of the hosts, which also failed. We then thought the issue could have been due to corrupted or unprocessable data in our Kafka stream, and tried various strategies to identify it or restore service. During that time, made the affected partition read-only and redirected the write traffic to our healthy partitions. Before the partition was marked as read-only, around 3% of datasets overall (which includes Honeycomb internal datasets) may have seen delayed data between September 28 at 16:17 UTC and 17:35 UTC (just over 1 hour). None of the data was lost, simply delayed for processing. All customer datasets are balanced across 3 partitions at a minimum, and gain more with higher ingest volume, so delayed data for that time period may have ranged from 30% on smaller datasets to less than 0.5% on larger ones. We tried restoring from older snapshots in case any bad data had corrupted disk data on the remaining hosts, until we realized that the process was failing to start because it had reached the maximum limit of open files allowed by the system. It had been set to a relatively high amount for the time, however, the amount of traffic that each partition receives combined with the number of columns (and therefore files) that the application holds open today is much higher than when that limit was set. We were able to hot-patch every running process with the higher limit in order to ensure no other partitions were affected - around 10x what it was before - but we waited to re-enable traffic to the affected partition in order to give our engineers time to thoroughly test a permanent fix that applied to all new instances at boot. As of today (September 30), we have re-enabled write traffic to the affected partition and have confirmed that the limit is applied to new instances. We’ve also added a signal to our alerting pipeline across the fleet for the number of open files as compared to the limit.
From Honeycomb’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.
Impact, status and times are the vendor’s own, as last archived. View on status.honeycomb.io (opens in new tab). Honeycomb profile · Recent Honeycomb incidents · Markdown · JSON