Changes / Lever / 500 Error when accessing Lever platform
Lever: Major incident: 500 Error when accessing Lever platform
Summary
Lever reported a major outage affecting most North America customers, with 500 errors blocking access to Lever Hire from 2025-02-18 15:12 UTC until 2025-02-18 21:09 UTC (5 h 57 min). Lever implemented a fix; its postmortem cited an unavailable database instance and compounding database-replica issues.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: 500 Error when accessing Lever platform
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-02-18 15:12 UTC
Before: not present
After: 2025-02-18 21:09 UTC
Before: not present
After: Global Data Center - LeverTRM - Hire, Hire
Before: not present
After: We have identified an issue where the 500 Error is received when accessing Lever, affecting most North America customers. We became aware of this issue at 9am EST. We will keep you updated as we continue working on this issue and await further information from our engineering teams.
Before: not present
After: Our team is still actively working to restore functionality to the Lever tool. We will continue to provide updates as they are made available.
Before: not present
After: A fix has been implemented for the issue causing the '500 error' messages. Customers should now be able to access hire.lever.co once more. We’re currently monitoring to ensure that there are no further issues, and we’ll send a final update to confirm that there have been no recurrences.
Before: not present
After: The issue where 500 errors were being encountered when attempting to access hire.lever.co has been resolved. There should be no further impact at this time, but please reach out to us at Support if any additional assistance is needed: https://help.lever.co/hc/en-us/requests/new
Before: not present
After: On the early morning of Feb 18, 2025 \(Pacific Time\), Lever product engineers were alerted by internal monitoring tools for a database instance being unavailable. Around 50% of customers may have initially experienced higher latency on Lever Hire and the Lever API.
A few hours later, compounding issues in the database replicas caused Lever Hire to be inaccessible for those ~50% of customers, for a few hours. The impacted customer accounts were unable to access Lever Hire and candidate-related Data API endpoints at all on Feb 18 from 5:55-10:05 PST \(the longest outage, ~4 hours\), 14:08-14:18, 14:48-14:52, 21:52-22:09; and Feb 19 from 00:06-00:14. Lever-hosted job sites continued to work for all customers.
Lever product engineers were engaged for investigation and troubleshooting pointed to database issues caused by unusual external Lever API load. The issue was resolved by:
* Rebuilding the affected database replicas
* Spreading Lever API load across additional database replicas
As part of this database recovery and mitigation measures, the Lever API also became inaccessible for all customers for a few hours. The database rebuild also caused some Lever Hire pipeline numbers and search results to be temporarily out of sync.
To mitigate this situation from occurring in the future and to reduce the risk of a future impact the following measures have been put into place:
* Per above, spreading Lever API load across additional database replicas
* Limiting database time for individual Lever API requests, to prevent a few individual requests from having a wider impact
* Optimizing database query performance
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: 500 Error when accessing Lever platform (this change) Lever reported a major outage affecting most North America customers, with 500 errors blocking access to Lever Hire from 2025-02-18 15:12 UTC until 2025-02-18 21:09 UTC (5 h 57 min). Lever implemented a fix; its postmortem cited an unavailable database instance and compounding database-replica issues. | Major incident |