Hex: Some users are experiencing issues accessing Hex projects
Some users could not access Hex projects during a critical Main site outage starting 2025-08-20 17:00 UTC, lasting 1 h 26 min and resolved at 2025-08-20 18:26 UTC. Hex implemented a fix and attributed the cascading failure to a feature flag, deployment timing during peak traffic, and inefficient database connection management.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Main site
Timeline of updates
- investigating
We are currently investigating this issue.
- investigating
We are continuing to investigate this issue.
- investigating
We are continuing to investigate this issue.
- identified
The issue has been identified and a fix is being implemented.
- monitoring
A fix has been implemented and we are monitoring the results.
- resolved
This incident has been resolved.
- postmortem
After fully analyzing the timeline of events, **we've confirmed that the root cause was a cascading failure triggered by the combination of a feature flag, deployment timing during peak traffic, and an inefficiency in our database connection management**. The issue began August 19 at 4:03 PM PDT, when a feature flag was updated to have clients query the server more frequently to eliminate a rare display issue in the client. It escalated August 20 at 10:00 AM PDT when we did a production deployment during peak traffic to add necessary support for a new feature release. This combination created a thundering herd problem where client reconnection attempts exponentially increased system pressure, saturating our database connections and triggering system-wide failures. Service was restored for most users at 10:52 PM PDT after manual intervention, and then fully restored at 11:26 AM EDT for all users after resetting backend pods. While this incident has been resolved, we are not satisfied by this sequence of events, from both the risk management perspective and the detection / response perspective. We are investing in the following mechanisms to improve: * We have increased baseline server capacity and reverted the change to the feature flag that increased database load. We are also optimizing database connection usage to handle traffic spikes via better pooling and reuse. * Our monitoring around system load and client connection patterns will be made more comprehensive, including proactive alerts on key server-side load and performance metrics for request processing, so we can identify and react to issues before they impact service. * We are improving our client-server reconnection protocol during releases to include rate limiting and exponential backoff, and investigating further advancements so we can handle deployment rollovers more gracefully without service disruption. * Until the improvements above are in place, we will be avoiding deployments during peak traffic times, unless absolutely necessary to resolve major issues with service stability.
Rewindex changes for this incident
- Critical incident: Some users are experiencing issues accessing Hex projects 2025-08-20 17:00 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Hex profile · Recent Hex incidents · Markdown · JSON