# Hex: Degraded kernel acquisition

- Impact (vendor’s label): critical
- Status: postmortem
- Started: 2025-05-13 14:38 UTC · Resolved: 2025-05-13 19:10 UTC · Duration: 4.5 h
- Updates: 7 · Affected components: Kernels
- [Vendor status page: https://stspg.io/m7r4dpkyqrwz](https://stspg.io/m7r4dpkyqrwz)
- [Company: Hex](https://rewindex.com/company/hex.md)

## Summary

Hex reported critical degradation of kernel acquisition from 2025-05-13 14:38 UTC to 2025-05-13 19:10 UTC, lasting 4 h 32 min. A fix was implemented; the postmortem attributed the incident to an AWS EBS storage issue backing its primary database replica, degrading replica performance and eventually backpressuring critical writes on the primary database. _(machine-written, sol)_
## Timeline of updates

- **investigating** 2025-05-13 14:38 UTC: We are currently investigating this issue.
- **investigating** 2025-05-13 16:27 UTC: We are continuing to investigate this issue.
- **monitoring** 2025-05-13 16:58 UTC: A fix has been implemented and we are monitoring the results.
- **identified** 2025-05-13 17:35 UTC: Some users are still experiencing issues as we implement a fix.
- **monitoring** 2025-05-13 18:09 UTC: A fix has been implemented and we are monitoring the results.
- **resolved** 2025-05-13 19:10 UTC: This incident has been resolved.
- **postmortem** 2025-05-21 20:20 UTC: After fully analyzing the timeline of events, \*\*we’ve confirmed that the root cause of the incident was an issue in the AWS EBS storage system backing our primary database replica\*\*. The issue began on May 12 11:00 PM PDT according to AWS, and the effect was gradual degradation of performance on the replica, eventually leading to a cascade where even our primary database got backpressured on critical write operations, which triggered the instability causing the incident. The AWS issue was resolved May 13 9:32 AM PDT, after we pushed a database configuration change that caused the EBS system software to reset. We then stabilized and cleaned up systems on our end before resolving the incident. While the root cause of this incident was on the AWS side, we are not satisfied that our detection and response were fast enough. We are investing in the following mechanisms to improve: \* We have automated testing to monitor provisioning of kernels, but it is combined with a larger testing suite that makes the signal less immediately actionable. We will be extracting this out as its own monitor so we can more quickly identify and react. \* Our database monitoring around latencies will be made more comprehensive, including the replica database, so we do not miss this early signal. \* We are improving our incident runbook to cover some of these checks earlier in the process so we zero in on the root cause more quickly.

## Rewindex changes for this incident

- 2025-05-13 14:38 UTC: [Critical incident: Degraded kernel acquisition](https://rewindex.com/change/i1a2d742fc57e82.md)


---
Canonical: https://rewindex.com/company/hex/incidents/3lkm1shzg6pj · Markdown: https://rewindex.com/company/hex/incidents/3lkm1shzg6pj.md · JSON: https://rewindex.com/api/v1/companies/hex/incidents/3lkm1shzg6pj · Generated 2026-10-18T12:35:21.569Z