Changes / Keeper / Duo Push in US region

Keeper: Major incident: Duo Push in US region

Summary

Summary pending. The diff below shows exactly what changed.

Major incidentLabel: degradation (model (Jev), 86%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: Duo Push in US region
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2023-12-13 17:29 UTC
AddedResolved time#5
Before: not present
After: 2023-12-13 18:19 UTC
AddedAffected components#6
Before: not present
After: End-User Client Applications - Keeper Browser Extensions, Keeper Browser Extensions, Keeper Infrastructure (US Data Center) - Keeper Web Vault (US), Keeper Web Vault (US)
AddedUpdateinvestigating#7
Before: not present
After: We are investigating intermittent Duo Push issues in the US region. Hard refreshing the browser window appears to mitigate the issue.
AddedUpdateresolved#8
Before: not present
After: Duo Push issues in the US region have been resolved. It may be necessary to reload the web vault page or restart the web browser.
AddedUpdatepostmortem#9
Before: not present
After: The Keeper engineering team has been researching the cause for recent Duo Push issues. Our investigation has resulted in the identification of a couple issues. The description is below: 1. The Duo Push services also have a direct dependency on the Keeper Push services which utilize websocket communication from the Keeper backend to the client device. 2. We determined that scaling additional instances of our push services was necessary to meet the growing usage of Duo and general push features across the Keeper end-user client apps 3. Scaling the backend instances immediately resolved the latency and the intermittent issues encountered by users. 4. In addition, we met with the Duo team, revisited our Duo integration code and we decided to rewrite some portions of the integration to reduce the number of dependencies and asynchronous processes. This is planned for release by December 20, 2023.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Major incident: Duo Push in US region (this change)Major incident

All Keeper changes → · Keeper profile · Markdown · JSON