Changes / Lever / Resumes and Offers Failing to Load
Lever: Major incident: Resumes and Offers Failing to Load
Summary
Lever reported a major incident: resumes failed to load, offers did not display, and manually adding resumes to opportunities produced errors. It ran from 2026-01-06 10:52 UTC to 2026-01-06 13:51 UTC, lasting 2 h 59 min and disrupting hiring workflows. A fix was implemented at 2026-01-06 12:47 UTC; no cause was stated.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Resumes and Offers Failing to Load
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2026-01-06 10:52 UTC
Before: not present
After: 2026-01-06 13:51 UTC
Before: not present
After: Global Data Center - LeverTRM - Hire, Hire
Before: not present
After: We are investigating an issue where resumes are not loading, offers are not displaying, and errors occur when manually adding resumes to opportunities. Our team is actively working to identify the root cause and restore normal functionality.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: We understand how important this functionality is to your hiring workflows, and our team moved quickly to investigate and address the issue. We can confirm this issue has been fully resolved, and our team has taken steps to ensure system stability moving forward.
## Customer Impact
* On January 6th, 2026, between 15:07 and 18:20 \(UTC\), users were **unable to upload resumes or documents** in Lever ATS.
* Upload attempts failed with an authorization-related error.
* Any new uploads during the incident window may result in references to non-existent files within the ATS. Unfortunately, these files are not recoverable.
* Other areas of the application continued to function normally.
## Root Cause
The incident was caused by an updated **configuration in Amazon S3 access policies** following a security-related infrastructure change.
As part of a security initiative, an existing IAM policy condition was updated in accordance with an AWS-recommended best practice. While this change was intended to improve security, the revised policy **did not behave as expected in our environment**, resulting in S3 denying permission for file uploads.
Because file uploads rely on this S3 permission, customers encountered authorization errors when uploading resumes and documents.
## Resolution
Once the issue was identified, the infrastructure team **reverted the policy change** to the previously functioning configuration.
After the rollback:
* File uploads resumed successfully
* Error rates returned to normal levels
* Customer impact was fully resolved
The system was verified as restored by confirming:
* Successful resume uploads from the application
* Elimination of S3 authorization errors in logs and dashboards
Invalid file reference cleanup
* Our Engineering teams are in the process of removing any invalid references to files that were not successfully uploaded during this time
## Preventative Actions
To reduce the likelihood of similar issues in the future, we are implementing the following improvements:
1. **Post-Deployment Monitoring**
* Enhanced monitoring after infrastructure and security-related changes in production
2. **Automated Alerts**
* Alerts for elevated file-upload failures and S3 authorization errors to reduce detection time
3. **Change Validation**
* Additional validation and testing of IAM policy changes, especially those involving AWS-recommended updates
4. **Documentation & Review**
* Improved documentation and cross-team review for security-driven infrastructure changes
These steps will help detect issues sooner and minimize customer impact in the event of future changes.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Resumes and Offers Failing to Load (this change) Lever reported a major incident: resumes failed to load, offers did not display, and manually adding resumes to opportunities produced errors. It ran from 2026-01-06 10:52 UTC to 2026-01-06 13:51 UTC, lasting 2 h 59 min and disrupting hiring workflows. A fix was implemented at 2026-01-06 12:47 UTC; no cause was stated. | Major incident |