Changes / Box / [Critical] Issues with multiple Box services
Box: Critical incident: [Critical] Issues with multiple Box services
Summary
Box reported a critical outage affecting Login/SSO, API, Box Notes and other services, with slowness and temporary unavailability. It ran from 2025-12-17 00:57 UTC to 2025-12-17 04:31 UTC (3 h 33 min); all affected services were restored. The postmortem linked the issue to an attempted rollback of a diagnostic change.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: [Critical] Issues with multiple Box services
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2025-12-17 00:57 UTC
Before: not present
After: 2025-12-17 04:31 UTC
Before: not present
After: Authentication (OAuth 2.0 / JWT), Box AI, Box Apps, Box Canvas, Box Doc Gen, Box Drive, Box Embed Widget, Box Notes - Web Application, Box Platform / API - Authentication (OAuth 2.0 / JWT), Box Platform / API - Box AI, Box Platform / API - Box Apps, Box Platform / API - Box Canvas, Box Platform / API - Box Doc Gen, Box Platform / API - Box Embed Widget, Box Platform / API - Box Shuttle, Box Platform / API - Box Sign, Box Platform / API - Content API, Box Platform / API - Content Preview, Box Platform / API - Search, Box Platform / API - Uploads/Downloads, Box Platform / API - Webhooks, Box Platform / API - Workflows and Automations, Box Shuttle, Box Sign, Box Web Application - Login/SSO, Content API, Content Preview, Desktop Applications - Box Drive, Login/SSO, Search, Uploads/Downloads, Web Application, Webhooks, Workflows and Automations
Before: not present
After: We are investigating an ongoing issue affecting Logins/SSO, API and Box Notes. We will provide more information as soon as it is available.
Before: not present
After: Our team has identified the underlying cause of this issue and is working to take remediating steps. We will provide additional updates as they become available.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: Our team has taken steps to remediate this issue and is seeing improvement for multiple Box services. We are continuing to monitor for any additional impact.
Before: not present
After: After further monitoring, this incident is now considered resolved. Our teams have validated that all affected services are restored to full functionality. Please contact Box Support at https://support.box.com/ if you continue to experience any issues.
Before: not present
After: We recently addressed issues affecting Box services**.** We would like to take the opportunity to further explain these issues and the steps we have taken to prevent them from happening in the future.
On December 16, 2025 between 4:25 PM PST and 7:23 PM PST, some users may have experienced slowness and temporary unavailability while working in Box. The issue occurred when we attempted to roll back a diagnostic change that was suspected of causing a minor degradation. The rollback unintentionally resulted in decreased capacity in our middleware service, leading to a degraded site experience. It also introduced a distributed deadlock condition that prolonged our incident response and mitigation efforts. Ultimately, we were able to resolve the deadlock and return to normal operations by temporarily throttling inbound traffic.
We have implemented measures to increase system capacity and minimize the possibility of a similar deadlock occurring in the future. We are also working on adding safeguards to avoid the rollback failure that served as the proximate cause of this incident.
**Analysis**
Between 11:11 AM PST and 1:38 PM PST, there was a production deployment of code that was aimed at gathering diagnostic information from our middleware service, in our ongoing efforts to improve system performance and resilience.
At 3:30 PM PST, we observed a brief, mostly imperceptible degradation and decided to roll back the change as a precaution. Due to failure in the rollback procedure, we observed a significantly reduced capacity in our middleware service that eventually led to a distributed deadlock condition between critical databases.
Box has made significant investments in reliability and mitigation measures. In this case, the issue was regional in scope and cross-region recovery capabilities are still in development. We therefore completed recovery by remediating the impacted region directly rather than executing a regional failover.
We restored middleware service capacity by 5:21 PM PST, but the deadlock condition persisted. After several lower-impact attempts to dislodge the deadlock were unsuccessful, we took a broader mitigation step and throttled all traffic at 6:54 PM PST. This resulted in a 26-minute full outage, after which systems stabilized and gradually recovered, with the site returning to full functionality at 7:23 PM PST.
**Corrective Actions**
Box has initiated the following corrective actions:
* Complete the planned capacity increase to reduce the chance of similar deadlocks and shorten recovery time if they occur
* Strengthen rollback safeguards so changes can be safely and reliably reverted, preventing a repeat of the rollback failure that triggered this incident
Thank you again for being a valued Box customer. We are continuously working to improve Box and want to make sure we are delivering the best in class product and user experience. Should you have any additional questions please do not hesitate to contact us.
Sincerely,
The Box Team
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: [Critical] Issues with multiple Box services (this change) Box reported a critical outage affecting Login/SSO, API, Box Notes and other services, with slowness and temporary unavailability. It ran from 2025-12-17 00:57 UTC to 2025-12-17 04:31 UTC (3 h 33 min); all affected services were restored. The postmortem linked the issue to an attempted rollback of a diagnostic change. | Major incident |