Changes / Hex / Degraded kernel acquisition
Hex: Critical incident: Degraded kernel acquisition
Summary
Hex reported critical degradation of kernel acquisition from 2025-05-13 14:38 UTC to 2025-05-13 19:10 UTC, lasting 4 h 32 min. A fix was implemented; the postmortem attributed the incident to an AWS EBS storage issue backing its primary database replica, degrading replica performance and eventually backpressuring critical writes on the primary database.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 99%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Degraded kernel acquisition
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2025-05-13 14:38 UTC
Before: not present
After: 2025-05-13 19:10 UTC
Before: not present
After: Kernels
Before: not present
After: We are currently investigating this issue.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: Some users are still experiencing issues as we implement a fix.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: After fully analyzing the timeline of events, **we’ve confirmed that the root cause of the incident was an issue in the AWS EBS storage system backing our primary database replica**. The issue began on May 12 11:00 PM PDT according to AWS, and the effect was gradual degradation of performance on the replica, eventually leading to a cascade where even our primary database got backpressured on critical write operations, which triggered the instability causing the incident. The AWS issue was resolved May 13 9:32 AM PDT, after we pushed a database configuration change that caused the EBS system software to reset. We then stabilized and cleaned up systems on our end before resolving the incident.
While the root cause of this incident was on the AWS side, we are not satisfied that our detection and response were fast enough. We are investing in the following mechanisms to improve:
* We have automated testing to monitor provisioning of kernels, but it is combined with a larger testing suite that makes the signal less immediately actionable. We will be extracting this out as its own monitor so we can more quickly identify and react.
* Our database monitoring around latencies will be made more comprehensive, including the replica database, so we do not miss this early signal.
* We are improving our incident runbook to cover some of these checks earlier in the process so we zero in on the root cause more quickly.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Degraded kernel acquisition (this change) Hex reported critical degradation of kernel acquisition from 2025-05-13 14:38 UTC to 2025-05-13 19:10 UTC, lasting 4 h 32 min. A fix was implemented; the postmortem attributed the incident to an AWS EBS storage issue backing its primary database replica, degrading replica performance and eventually backpressuring critical writes on the primary database. | Major incident |