Changes / 1Password / Degraded performance when accessing 1Password
1Password: Major incident: Degraded performance when accessing 1Password
Summary
1Password reported a major degradation affecting its web interface and APIs for all customers in the US region, including SSO, sign-in, sign-up and CLI. The incident ran from 2025-09-26 16:51 UTC to 2025-09-26 20:24 UTC (3 h 33 min); a deployed mitigation addressed the issue.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
What changed
Before: not present
After: Degraded performance when accessing 1Password
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-09-26 16:51 UTC
Before: not present
After: 2025-09-26 20:24 UTC
Before: not present
After: 1Password Connect, Admin console, Billing, Command Line Interface (CLI), Multi-factor Authentication (MFA), Sign in, Sign up, SSO (Single Sign On), Syncing items between your devices, USA/Global - 1Password Connect, USA/Global - Admin console, USA/Global - Billing, USA/Global - Command Line Interface (CLI), USA/Global - Multi-factor Authentication (MFA), USA/Global - Sign in, USA/Global - Sign up, USA/Global - SSO (Single Sign On), USA/Global - Syncing items between your devices
Before: not present
After: We are actively investigating an issue where customers may be experiencing degraded performance and slowdowns when accessing 1Password.
Before: not present
After: The engineering team is restarting key systems in an attempt to alleviate slowdowns.
Before: not present
After: Overall service is still degraded, but experiencing periods of improvement.
Before: not present
After: The engineering team is continuing to investigate and actively managing service degradation. Customers may still experience periods of slowdowns when accessing the 1Password service.
Before: not present
After: The engineering team is actively managing degraded performance and simultaneously attempting to identify root cause. Customers may still experience some slowdowns when accessing 1Password online.
Before: not present
After: The engineering team is actively managing degraded performance and continuing attempts to identify root cause. Customers may still experience some slowdowns when accessing 1Password online.
Before: not present
After: Engineering teams have identified a potential cause and are actively managing customer impact. Customers may see some slowdowns, but due to the active management those slowdowns are unlikely.
Before: not present
After: Engineering teams are testing a mitigation in our non-production environments to verify before rolling it out to our production environments. Customer impact is being actively managed and slowdowns should be rare if they occur at all.
Before: not present
After: The engineering team has identified the issue and we are continuing to test a mitigation in our test environment before rolling it out to production. Customer impact is being actively managed and slowdowns should be increasingly rare.
Before: not present
After: The engineering team is continuing to test a mitigation in our test environment before deploying it to production. Customer impact is being actively managed and instances of slowdowns should continue to decrease.
Before: not present
After: The engineering team is rolling out a mitigation in our production environment.
Before: not present
After: The engineering team deployed a mitigation, which has addressed the issue. We will continue to monitor performance.
Before: not present
After: This incident has been resolved. We will publish a post-mortem as soon as we complete it.
Before: not present
After: # Incident Postmortem - Degraded performance when accessing 1Password
**Date of Incident:** 2025-09-26
**Time of Incident:** 4:20pm UTC - 5:39pm UTC
**Service\(s\) Affected:** SSO, Web Sign In, Sign Up, Web Interface, CLI
**Impact Duration:** ~60 minutes
## Summary
On September 26, 2025 at 4:20 UTC 1Password’s web interface and APIs experienced degraded performance for all customers in the US region. This was not a result of a security incident and customer data was not affected.
## Impact on Customers
During the duration of the incident:
* **Web interface, Administration:** Customers experienced delays when accessing the 1Password web interface.
* **Single Sign-on \(SSO\), Multi-factor Authentication \(MFA\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to login.
* **Command Line Interface \(CLI\):** CLI users faced increased latency and timeouts when attempting to access our web APIs.
* **Browser Extension:** Users requiring web interface authentication experienced delays or failures.
* **Number of Affected Customers \(approximate\):** ~30%
* **Geographic Regions Affected:** [1password.com](http://1password.com) \(US/Global\)
## What Happened?
At 4:20PM UTC and 5 PM UTC There were traffic bursts which caused extra load on one of our caches. This cache was under-provisioned to handle that spike of activity, which resulted in it exhausting available CPU. This caused cascading errors/latency which manifested in slow and failed requests.
* **Timeline of Events \(UTC\):**
* 2025-09-26 4:20pm: Spike in customer traffic began
* 2025-09-26 4:29pm: Automated monitoring detects increased errors and latency
* 2025-09-26 4:35pm: The team activates our incident protocol and begins investigation
* 2025-09-26 4:58pm: The team decides to restart application servers
* 2025-09-26 5:00pm: The servers have been restarted, service is still degraded, as a second traffic burst begins
* 2025-09-26 5:18pm: Service starts to improve
* 2025-09-26 5:25pm: The team detects increased load for the second time
* 2025-09-26 5:33pm: The team restarts application servers again
* 2025-09-26 5:39pm: Service is back to normal, team continues to investigate
* 2025-09-26 7:26pm: Team has found the issue, and proceeds to upgrade cache instance size
* 2025-09-26 7:49pm: Cache upgrade completed successfully
* 2025-09-26 7:50pm: Team continues to monitor, performance has returned to nominal levels
* 2025-09-26 8:24pm: Incident is marked as resolved
* **Root Cause Analysis:**
A code library installed in July introduced latency issues for cache connections. Authentication operations weren't properly rate-limited, allowing large traffic influxes. During peak traffic periods, the cache infrastructure was operating near maximum CPU capacity. The incident occurred when a burst of authentication traffic pushed the cache CPU utilization to 100%. The increased latency and CPU usage together directly caused the incident.
* **Contributing Factors:**
* Latency increase due to cache library version upgrade
* Inadequate rate limiting allowed traffic bursts to go unchecked
* Cache instance size is under-provisioned
## How Was It Resolved?
* **Mitigation Steps:** Restarting application servers temporarily mitigated the latency and errors, but the problems returned when traffic spiked again.
* **Resolution Steps:** Increasing the instance size for the cache resolved the issue.
* **Verification of Resolution:** The incident team tested the upgrade in a staging deployment before executing it in production. They then monitored metrics to confirm the system returned to normal levels.
## What We Are Doing to Prevent Future Incidents
* **Improve capacity planning for cache:** We will ensure our internal infrastructure is properly sized to handle current traffic volumes and accommodate future growth. We'll implement regular resource evaluations to maintain adequate capacity as our traffic increases. We will also implement proactive alerting systems that notify our teams when resource utilization approaches critical thresholds.
* **Update library to a more performant version:** We will upgrade our caching library to the latest stable version to eliminate the current latency issues.
* **Improve rate limiting for operations that triggered the traffic burst:** Enhancing our rate limiting system will significantly improve our ability to handle future traffic bursts.
* **Timeline for Implementation:** Observability improvements have already been implemented, and we will complete the remaining work by the end of Q1, 2026.
## Next Steps and Communication
No action is required from our customers at this time.
We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.
Sincerely,
The 1Password Team
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Degraded performance when accessing 1Password (this change) 1Password reported a major degradation affecting its web interface and APIs for all customers in the US region, including SSO, sign-in, sign-up and CLI. The incident ran from 2025-09-26 16:51 UTC to 2025-09-26 20:24 UTC (3 h 33 min); a deployed mitigation addressed the issue. | Major incident |
All 1Password changes → · 1Password profile · Markdown · JSON