Changes / Lever / Offer Letter Generation Failures
Lever: Critical incident: Offer Letter Generation Failures
Summary
Lever reported a critical outage affecting offer-letter generation and delivery in Global Data Center - LeverTRM - Hire, Hire +, starting 2026-07-14 14:47 UTC and resolved 2026-07-14 23:50 UTC (9 h 2 min). The postmortem reported intermittent document/resume preview failures caused by a failed conversion step; no data loss occurred.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Offer Letter Generation Failures
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2026-07-14 14:47 UTC
Before: not present
After: 2026-07-14 23:50 UTC
Before: not present
After: Global Data Center - LeverTRM - Hire, Hire
Before: not present
After: Issue:
We are investigating what appears to be a platform-wide issue in which offer letters fail to be sent.
Customer Impact:
Recruiters are currently unable to generate or deliver offer emails to candidates.
Current Status:
Engineering is actively investigating. No workaround is available at this time.
Before: not present
After: Issue: We identified a platform-wide issue in which offer letters failed to be sent.
Customer Impact: Recruiters were unable to generate or deliver offer emails to candidates.
Current Status: The immediate issue has been mitigated. We are actively monitoring to confirm full resolution and stability.
Before: not present
After: Issue: We identified a platform-wide issue in which offer letters failed to be sent.
Customer Impact: Recruiters were unable to generate or deliver offer emails to candidates.
Status: Resolved. An RCA will be available within 7-10 business days.
Before: not present
After: **Incident Summary**
Between approximately July 13 and July 14, 2026, some customers were intermittently unable to send offer letters, and some document/resume previews failed to generate. While source files were uploaded successfully, the conversion step required for previews and offer generation failed. No data loss occurred.
**Detection**
The issue was identified on July 14, 2026, at approximately 10:43 AM ET, following reports from customers to our Support team. Engineering confirmed the impact through monitoring, which showed a steady increase in process counts within the document-converter service until it reached a hard system limit, causing subsequent tasks to fail.
**Root Cause**
The incident was caused by a "process leak" introduced during a recent platform upgrade. A change in how the system handles document conversions meant that each conversion started a new background process that was never properly closed. Over several days, these background processes accumulated until the system reached its maximum capacity. Once this limit was hit, the service could no longer start the new processes required to generate offer letters or document previews.
**Resolution**
* **Initial Mitigation:** At 11:20 AM ET on July 14, engineers restarted the affected service pods. This cleared the accumulated background processes and immediately restored the ability for customers to send offer letters and view previews.
* **Permanent Fix:** A code update was developed and deployed at 8:43 PM ET on July 14. This fix ensures that all background processes are correctly terminated and cleaned up after every conversion, preventing any future accumulation.
**Preventative Measures**
* **Code Improvements:** Implemented a robust process-management system \(using a "reaper" process\) to ensure all background tasks are automatically closed upon completion.
* **Enhanced Monitoring:** New automated alerts have been configured to notify the engineering team if process usage exceeds 50% of the system limit. This will allow us to identify and resolve similar resource leaks days before they impact customers.
* **Backlog Processing:** A separate initiative was completed to identify and reprocess any document or resume previews that failed to generate during the outage window to ensure all previews are available to users.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Offer Letter Generation Failures (this change) Lever reported a critical outage affecting offer-letter generation and delivery in Global Data Center - LeverTRM - Hire, Hire +, starting 2026-07-14 14:47 UTC and resolved 2026-07-14 23:50 UTC (9 h 2 min). The postmortem reported intermittent document/resume preview failures caused by a failed conversion step; no data loss occurred. | Major incident |