# 1Password: Degraded performance when accessing 1Password - Impact (vendor’s label): major - Status: postmortem - Started: 2025-09-26 16:51 UTC · Resolved: 2025-09-26 20:24 UTC · Duration: 3.5 h - Updates: 14 · Affected components: Command Line Interface (CLI), USA/Global - Command Line Interface (CLI), Sign up, USA/Global - Sign up, Syncing items between your devices, USA/Global - Syncing items between your devices, 1Password Connect, USA/Global - 1Password Connect, Multi-factor Authentication (MFA), USA/Global - Multi-factor Authentication (MFA), Sign in, USA/Global - Sign in, Billing, USA/Global - Billing, SSO (Single Sign On), USA/Global - SSO (Single Sign On), Admin console, USA/Global - Admin console - [Vendor status page: https://stspg.io/0bw2wt81svs7](https://stspg.io/0bw2wt81svs7) - [Company: 1Password](https://rewindex.com/company/1password.md) ## Summary 1Password reported a major degradation affecting its web interface and APIs for all customers in the US region, including SSO, sign-in, sign-up and CLI. The incident ran from 2025-09-26 16:51 UTC to 2025-09-26 20:24 UTC (3 h 33 min); a deployed mitigation addressed the issue. _(machine-written, sol)_ ## Timeline of updates - **investigating** 2025-09-26 16:51 UTC: We are actively investigating an issue where customers may be experiencing degraded performance and slowdowns when accessing 1Password. - **investigating** 2025-09-26 17:06 UTC: The engineering team is restarting key systems in an attempt to alleviate slowdowns. - **investigating** 2025-09-26 17:25 UTC: Overall service is still degraded, but experiencing periods of improvement. - **investigating** 2025-09-26 17:41 UTC: The engineering team is continuing to investigate and actively managing service degradation. Customers may still experience periods of slowdowns when accessing the 1Password service. - **investigating** 2025-09-26 17:57 UTC: The engineering team is actively managing degraded performance and simultaneously attempting to identify root cause. Customers may still experience some slowdowns when accessing 1Password online. - **investigating** 2025-09-26 18:11 UTC: The engineering team is actively managing degraded performance and continuing attempts to identify root cause. Customers may still experience some slowdowns when accessing 1Password online. - **investigating** 2025-09-26 18:32 UTC: Engineering teams have identified a potential cause and are actively managing customer impact. Customers may see some slowdowns, but due to the active management those slowdowns are unlikely. - **investigating** 2025-09-26 18:44 UTC: Engineering teams are testing a mitigation in our non-production environments to verify before rolling it out to our production environments. Customer impact is being actively managed and slowdowns should be rare if they occur at all. - **identified** 2025-09-26 18:57 UTC: The engineering team has identified the issue and we are continuing to test a mitigation in our test environment before rolling it out to production. Customer impact is being actively managed and slowdowns should be increasingly rare. - **identified** 2025-09-26 19:13 UTC: The engineering team is continuing to test a mitigation in our test environment before deploying it to production. Customer impact is being actively managed and instances of slowdowns should continue to decrease. - **identified** 2025-09-26 19:28 UTC: The engineering team is rolling out a mitigation in our production environment. - **monitoring** 2025-09-26 19:53 UTC: The engineering team deployed a mitigation, which has addressed the issue. We will continue to monitor performance. - **resolved** 2025-09-26 20:24 UTC: This incident has been resolved. We will publish a post-mortem as soon as we complete it. - **postmortem** 2025-10-03 19:24 UTC: \# Incident Postmortem - Degraded performance when accessing 1Password \*\*Date of Incident:\*\* 2025-09-26 \*\*Time of Incident:\*\* 4:20pm UTC - 5:39pm UTC \*\*Service\\(s\\) Affected:\*\* SSO, Web Sign In, Sign Up, Web Interface, CLI \*\*Impact Duration:\*\* ~60 minutes \#\# Summary On September 26, 2025 at 4:20 UTC 1Password’s web interface and APIs experienced degraded performance for all customers in the US region. This was not a result of a security incident and customer data was not affected. \#\# Impact on Customers During the duration of the incident: \* \*\*Web interface, Administration:\*\* Customers experienced delays when accessing the 1Password web interface. \* \*\*Single Sign-on \\(SSO\\), Multi-factor Authentication \\(MFA\\):\*\* Users with SSO or MFA enabled experienced delays, and in some cases failures to login. \* \*\*Command Line Interface \\(CLI\\):\*\* CLI users faced increased latency and timeouts when attempting to access our web APIs. \* \*\*Browser Extension:\*\* Users requiring web interface authentication experienced delays or failures. \* \*\*Number of Affected Customers \\(approximate\\):\*\* ~30% \* \*\*Geographic Regions Affected:\*\* \[1password.com\](http://1password.com) \\(US/Global\\) \#\# What Happened? At 4:20PM UTC and 5 PM UTC There were traffic bursts which caused extra load on one of our caches. This cache was under-provisioned to handle that spike of activity, which resulted in it exhausting available CPU. This caused cascading errors/latency which manifested in slow and failed requests. \* \*\*Timeline of Events \\(UTC\\):\*\* \* 2025-09-26 4:20pm: Spike in customer traffic began \* 2025-09-26 4:29pm: Automated monitoring detects increased errors and latency \* 2025-09-26 4:35pm: The team activates our incident protocol and begins investigation \* 2025-09-26 4:58pm: The team decides to restart application servers \* 2025-09-26 5:00pm: The servers have been restarted, service is still degraded, as a second traffic burst begins \* 2025-09-26 5:18pm: Service starts to improve \* 2025-09-26 5:25pm: The team detects increased load for the second time \* 2025-09-26 5:33pm: The team restarts application servers again \* 2025-09-26 5:39pm: Service is back to normal, team continues to investigate \* 2025-09-26 7:26pm: Team has found the issue, and proceeds to upgrade cache instance size \* 2025-09-26 7:49pm: Cache upgrade completed successfully \* 2025-09-26 7:50pm: Team continues to monitor, performance has returned to nominal levels \* 2025-09-26 8:24pm: Incident is marked as resolved \* \*\*Root Cause Analysis:\*\* A code library installed in July introduced latency issues for cache connections. Authentication operations weren't properly rate-limited, allowing large traffic influxes. During peak traffic periods, the cache infrastructure was operating near maximum CPU capacity. The incident occurred when a burst of authentication traffic pushed the cache CPU utilization to 100%. The increased latency and CPU usage together directly caused the incident. \* \*\*Contributing Factors:\*\* \* Latency increase due to cache library version upgrade \* Inadequate rate limiting allowed traffic bursts to go unchecked \* Cache instance size is under-provisioned \#\# How Was It Resolved? \* \*\*Mitigation Steps:\*\* Restarting application servers temporarily mitigated the latency and errors, but the problems returned when traffic spiked again. \* \*\*Resolution Steps:\*\* Increasing the instance size for the cache resolved the issue. \* \*\*Verification of Resolution:\*\* The incident team tested the upgrade in a staging deployment before executing it in production. They then monitored metrics to confirm the system returned to normal levels. \#\# What We Are Doing to Prevent Future Incidents \* \*\*Improve capacity planning for cache:\*\* We will ensure our internal infrastructure is properly sized to handle current traffic volumes and accommodate future growth. We'll implement regular resource evaluations to maintain adequate capacity as our traffic increases. We will also implement proactive alerting systems that notify our teams when resource utilization approaches critical thresholds. \* \*\*Update library to a more performant version:\*\* We will upgrade our caching library to the latest stable version to eliminate the current latency issues. \* \*\*Improve rate limiting for operations that triggered the traffic burst:\*\* Enhancing our rate limiting system will significantly improve our ability to handle future traffic bursts. \* \*\*Timeline for Implementation:\*\* Observability improvements have already been implemented, and we will complete the remaining work by the end of Q1, 2026. \#\# Next Steps and Communication No action is required from our customers at this time. ‌ We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team ## Rewindex changes for this incident - 2025-09-26 16:51 UTC: [Major incident: Degraded performance when accessing 1Password](https://rewindex.com/change/ifdd0c2e4c991da.md) --- Canonical: https://rewindex.com/company/1password/incidents/z20sn66ynbfl · Markdown: https://rewindex.com/company/1password/incidents/z20sn66ynbfl.md · JSON: https://rewindex.com/api/v1/companies/1password/incidents/z20sn66ynbfl · Generated 2026-10-18T12:22:14.004Z