rewindex

CompaniesKeeperIncidents

Keeper: Vault login errors

No impact listedStatus: postmortemStarted Resolved (-280 min)2 updatesstspg.io (opens in new tab)

Timeline of updates

  1. resolved

    Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.

  2. postmortem

    Incident Summary

    The Keeper DevOps team was alerted to API errors in the US region. Upon investigation, we identified a high number of database deadlocks affecting performance. To mitigate the immediate impact, we restarted the affected writer instance.

    Root Cause

    Historically, similar issues were linked to specific queries impacting MySQL 8.x updates on the RDS Aurora platform. In this case, we discovered that a utility designed to monitor deadlocks and capture diagnostic data was inadvertently contributing to the problem. A known bug in the MySQL minor version used in production caused our data-gathering query—intended to analyze deadlocks—to trigger additional locking, exacerbating the issue.

    Resolution & Mitigation

    • We promptly disabled the problematic monitoring utility and applied database parameter optimizations in collaboration with the AWS team to enhance performance under peak load.
    • These changes were fully implemented within 24 hours, and no further excessive row locks have been observed.
    • Our DBA team is actively working with backend engineers to further optimize write operations for improved performance.
    • A backend code update with additional optimizations is scheduled for release the week of March 17.

    We appreciate your patience and will continue to monitor and enhance system performance to prevent similar issues in the future.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Cite this

Status incident (vendor impact label: none) posted by Keeper, started 2025-03-11 04:40 UTC, resolved after -280 min, 2 update(s).

Title
Vault login errors
Vendor
Keeper
Date
URL
https://rewindex.com/company/keeper/incidents/7335ggg1z9mr

Keeper profileRecent Keeper incidentsAll outages on March 11, 2025Keeper in March 2025MarkdownJSON