Changes / Hex / Some users are experiencing issues accessing Hex projects
Hex: Critical incident: Some users are experiencing issues accessing Hex projects
Summary
Some users could not access Hex projects during a critical Main site outage starting 2025-08-20 17:00 UTC, lasting 1 h 26 min and resolved at 2025-08-20 18:26 UTC. Hex implemented a fix and attributed the cascading failure to a feature flag, deployment timing during peak traffic, and inefficient database connection management.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Some users are experiencing issues accessing Hex projects
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2025-08-20 17:00 UTC
Before: not present
After: 2025-08-20 18:26 UTC
Before: not present
After: Main site
Before: not present
After: We are currently investigating this issue.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: The issue has been identified and a fix is being implemented.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: After fully analyzing the timeline of events, **we've confirmed that the root cause was a cascading failure triggered by the combination of a feature flag, deployment timing during peak traffic, and an inefficiency in our database connection management**. The issue began August 19 at 4:03 PM PDT, when a feature flag was updated to have clients query the server more frequently to eliminate a rare display issue in the client. It escalated August 20 at 10:00 AM PDT when we did a production deployment during peak traffic to add necessary support for a new feature release. This combination created a thundering herd problem where client reconnection attempts exponentially increased system pressure, saturating our database connections and triggering system-wide failures. Service was restored for most users at 10:52 PM PDT after manual intervention, and then fully restored at 11:26 AM EDT for all users after resetting backend pods.
While this incident has been resolved, we are not satisfied by this sequence of events, from both the risk management perspective and the detection / response perspective. We are investing in the following mechanisms to improve:
* We have increased baseline server capacity and reverted the change to the feature flag that increased database load. We are also optimizing database connection usage to handle traffic spikes via better pooling and reuse.
* Our monitoring around system load and client connection patterns will be made more comprehensive, including proactive alerts on key server-side load and performance metrics for request processing, so we can identify and react to issues before they impact service.
* We are improving our client-server reconnection protocol during releases to include rate limiting and exponential backoff, and investigating further advancements so we can handle deployment rollovers more gracefully without service disruption.
* Until the improvements above are in place, we will be avoiding deployments during peak traffic times, unless absolutely necessary to resolve major issues with service stability.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Some users are experiencing issues accessing Hex projects (this change) Some users could not access Hex projects during a critical Main site outage starting 2025-08-20 17:00 UTC, lasting 1 h 26 min and resolved at 2025-08-20 18:26 UTC. Hex implemented a fix and attributed the cascading failure to a feature flag, deployment timing during peak traffic, and inefficient database connection management. | Major incident |