# Box: Critical incident: [Critical] Issues with multiple Box services

**Summary:** Box reported a critical outage affecting Login/SSO, API, Box Notes and other services, with slowness and temporary unavailability. It ran from 2025-12-17 00:57 UTC to 2025-12-17 04:31 UTC (3 h 33 min); all affected services were restored. The postmortem linked the issue to an attempted rollback of a diagnostic change. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2025-12-17 00:57 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/6q49d8qwrq6b](https://stspg.io/6q49d8qwrq6b)
- [Company: Box](https://rewindex.com/company/box.md)
- Versions: before (none: first archived) → after incident\_version:93/48t1p9xtd4tz@42e0fdeaddb0
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ [Critical] Issues with multiple Box services
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2025-12-17 00:57 UTC
```

**5. Added: Resolved time**

```diff
+ 2025-12-17 04:31 UTC
```

**6. Added: Affected components**

```diff
+ Authentication (OAuth 2.0 / JWT), Box AI, Box Apps, Box Canvas, Box Doc Gen, Box Drive, Box Embed Widget, Box Notes - Web Application, Box Platform / API - Authentication (OAuth 2.0 / JWT), Box Platform / API - Box AI, Box Platform / API - Box Apps, Box Platform / API - Box Canvas, Box Platform / API - Box Doc Gen, Box Platform / API - Box Embed Widget, Box Platform / API - Box Shuttle, Box Platform / API - Box Sign, Box Platform / API - Content API, Box Platform / API - Content Preview, Box Platform / API - Search, Box Platform / API - Uploads/Downloads, Box Platform / API - Webhooks, Box Platform / API - Workflows and Automations, Box Shuttle, Box Sign, Box Web Application - Login/SSO, Content API, Content Preview, Desktop Applications - Box Drive, Login/SSO, Search, Uploads/Downloads, Web Application, Webhooks, Workflows and Automations
```

**7. Added: Update** (investigating) · 2025-12-17 00:57 UTC

```diff
+ We are investigating an ongoing issue affecting Logins/SSO, API  and Box Notes.  We will provide more information as soon as it is available.
```

**8. Added: Update** (identified) · 2025-12-17 01:19 UTC

```diff
+ Our team has identified the underlying cause of this issue and is working to take remediating steps. We will provide additional updates as they become available.
```

**9. Added: Update** (identified) · 2025-12-17 01:42 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**10. Added: Update** (identified) · 2025-12-17 02:36 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**11. Added: Update** (identified) · 2025-12-17 03:17 UTC

```diff
+ We are continuing to work on a fix for this issue.
```

**12. Added: Update** (monitoring) · 2025-12-17 03:42 UTC

```diff
+ Our team has taken steps to remediate this issue and is seeing improvement for multiple Box services. We are continuing to monitor for any additional impact.
```

**13. Added: Update** (resolved) · 2025-12-17 04:31 UTC

```diff
+ After further monitoring, this incident is now considered resolved. Our teams have validated that all affected services are restored to full functionality. Please contact Box Support at https://support.box.com/ if you continue to experience any issues.
```

**14. Added: Update** (postmortem) · 2025-12-18 01:45 UTC

```diff
+ We recently addressed issues affecting Box services**.** We would like to take the opportunity to further explain these issues and the steps we have taken to prevent them from happening in the future. On December 16, 2025 between 4:25 PM PST and 7:23 PM PST, some users may have experienced slowness and temporary unavailability while working in Box. The issue occurred when we attempted to roll back a diagnostic change that was suspected of causing a minor degradation. The rollback unintentionally resulted in decreased capacity in our middleware service, leading to a degraded site experience. It also introduced a distributed deadlock condition that prolonged our incident response and mitigation efforts. Ultimately, we were able to resolve the deadlock and return to normal operations by temporarily throttling inbound traffic. We have implemented measures to increase system capacity and minimize the possibility of a similar deadlock occurring in the future. We are also working on adding safeguards to avoid the rollback failure that served as the proximate cause of this incident. **Analysis** Between 11:11 AM PST and 1:38 PM PST, there was a production deployment of code that was aimed at gathering diagnostic information from our middleware service, in our ongoing efforts to improve system performance and resilience. At 3:30 PM PST, we observed a brief, mostly imperceptible degradation and decided to roll back the change as a precaution. Due to failure in the rollback procedure, we observed a significantly reduced capacity in our middleware service that eventually led to a distributed deadlock condition between critical databases. Box has made significant investments in reliability and mitigation measures. In this case, the issue was regional in scope and cross-region recovery capabilities are still in development. We therefore completed recovery by remediating the impacted region directly rather than executing a regional failover. We restored middleware service capacity by 5:21 PM PST, but the deadlock condition persisted. After several lower-impact attempts to dislodge the deadlock were unsuccessful, we took a broader mitigation step and throttled all traffic at 6:54 PM PST. This resulted in a 26-minute full outage, after which systems stabilized and gradually recovered, with the site returning to full functionality at 7:23 PM PST. **Corrective Actions** Box has initiated the following corrective actions: * Complete the planned capacity increase to reduce the chance of similar deadlocks and shorten recovery time if they occur * Strengthen rollback safeguards so changes can be safely and reliably reverted, preventing a repeat of the rollback failure that triggered this incident Thank you again for being a valued Box customer.  We are continuously working to improve Box and want to make sure we are delivering the best in class product and user experience. Should you have any additional questions please do not hesitate to contact us. Sincerely, The Box Team
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2025-12-17 00:57 UTC | Critical incident: \[Critical\] Issues with multiple Box services (this change) | outage |

All Box changes: [timeline](https://rewindex.com/company/box/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i3d8e3799fd4b8a · Markdown: https://rewindex.com/change/i3d8e3799fd4b8a.md · JSON: https://rewindex.com/api/v1/changes/i3d8e3799fd4b8a · Generated 2026-10-18T01:30:00.422Z