Changes / Lever / Lever -hosted job site down
Lever: Critical incident: Lever -hosted job site down
Summary
Lever-hosted job sites and the Postings API were down in a critical incident starting 2025-10-03 21:02 UTC and resolved at 2025-10-03 23:01 UTC (1 h 58 min). Lever attributed the outage to an unexpected pattern of incoming requests and reported implementing a fix at 10-03 22:09 UTC.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Lever -hosted job site down
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2025-10-03 21:02 UTC
Before: not present
After: 2025-10-03 23:01 UTC
Before: not present
After: Career Site, Global Data Center - LeverTRM - Career Site, Global Data Center - LeverTRM - Hire, Hire
Before: not present
After: At 2:04pm PDT we identified an issue causing Lever-hosted job sites and Postings API to be down. Users will notice a looping loading wheel and the page may timeout while loading. We are actively investigating the issue to implement a fix. Additional updates will be provided as soon as a resolution is in place.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: We have implemented a fix for the issue. All Lever-hosted job site services should be available. We are continuing to monitor.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
Before: not present
After: ## **Incident RCA: Job Site Outage**
**Incident Date:** October 3, 2025
**Start Time:** 4:51 PM ET
**End Time:** 6:00 PM ET
### **Impact Summary**
Between 4:51 PM and 6:00 PM ET on October 3, 2025, users experienced degraded performance and eventual unavailability across multiple job boards.
### **Root Cause**
The outage was triggered by an unexpected pattern of incoming requests that, while not unusually high in volume, each initiated resource-intensive queries. This combination of request frequency and prolonged query execution overwhelmed backend systems, leading to degraded performance and service unavailability. The traffic resembled automated crawling behavior, which our systems were not optimized to handle at that level of query complexity and concurrency.
### **Resolution**
To reduce server load and restore service availability, the impacted job site was temporarily disabled. This action immediately alleviated pressure on backend systems and allowed all job boards to recover. The site remained offline over the weekend and was re-enabled on October 6, 2025, after Engineering confirmed that the irregular traffic had subsided.
### **Preventive Actions**
To prevent similar incidents in the future, the following measures have been implemented:
* **Enhanced Caching:** Improvements to our job site caching mechanisms now limit the ability of similar request patterns to generate excessive server queries.
* **Updated Crawling Guidelines:** We have revised our web crawling directives to encourage automated crawlers to reduce the frequency of their requests. This change helps maintain system stability by minimizing the impact of high-volume automated traffic from compliant sources.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Lever -hosted job site down (this change) Lever-hosted job sites and the Postings API were down in a critical incident starting 2025-10-03 21:02 UTC and resolved at 2025-10-03 23:01 UTC (1 h 58 min). Lever attributed the outage to an unexpected pattern of incoming requests and reported implementing a fix at 10-03 22:09 UTC. | Major incident |