Changes / Keeper / Vault login issues in US

Keeper: Critical incident: Vault login issues in US

Summary

Keeper reported critical vault login issues affecting Keeper Web Vault (US) and Keeper Admin Console (US), from 2023-09-19 17:37 UTC to 2023-09-19 17:43 UTC (6 min). Its postmortem described intermittent API errors and identified bugs in the desktop app and backend API; DevOps resolved the issue.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 92%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Vault login issues in US
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2023-09-19 17:37 UTC
AddedResolved time#5
Before: not present
After: 2023-09-19 17:43 UTC
AddedAffected components#6
Before: not present
After: Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Admin Console (US), Keeper Infrastructure (US Data Center) - Keeper Web Vault (US), Keeper Web Vault (US)
AddedUpdateinvestigating#7
Before: not present
After: We are investigating vault login issues in the US data center.
AddedUpdateresolved#8
Before: not present
After: Vault login issues in the US has been resolved. We will update this statuspage event with additional details.
AddedUpdatepostmortem#9
Before: not present
After: Our team received notification of API errors at 10:20AM PST. Between 10:20AM and 10:50AM the errors were intermittent and causing vault login errors in the US data center. The DevOps team resolved the issue fully at 10:53AM PST. After resolution of the issue, the engineering team researched the cause and found that a bug in the desktop app and also a related bug down the chain of processes in the backend API caused some client devices to generate a high number of full-sync API calls. We have identified the root cause and a patch is being applied to both Desktop App and Backend API components over the next 24 hours.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Critical incident: Vault login issues in US (this change)
Keeper reported critical vault login issues affecting Keeper Web Vault (US) and Keeper Admin Console (US), from 2023-09-19 17:37 UTC to 2023-09-19 17:43 UTC (6 min). Its postmortem described intermittent API errors and identified bugs in the desktop app and backend API; DevOps resolved the issue.
Major incident

All Keeper changes → · Keeper profile · Markdown · JSON