Changes / Doppler / API unavailable due to contention over database locking
Doppler: Major incident: API unavailable due to contention over database locking
Summary
Doppler reported a major API outage caused by database lock contention on its Workplace table. Recorded start: 2021-10-01 19:34 UTC; resolution: 2021-10-01 09:00 UTC; computed duration unknown (vendor reported 11 minutes). Service recovered when locks expired; Doppler refactored the affected component to avoid locking.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: API unavailable due to contention over database locking
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2021-10-01 19:34 UTC
Before: not present
After: 2021-10-01 09:00 UTC
Before: not present
After: Our API experienced an 11 minute outage due to database contention over locking our Workplace table. The issue resolved itself once the locks expired. We've refactored the affected component to achieve the desired result without locking. We'll be exploring a more holistic fix to prevent this issue from occurring in other components.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: API unavailable due to contention over database locking (this change) Doppler reported a major API outage caused by database lock contention on its Workplace table. Recorded start: 2021-10-01 19:34 UTC; resolution: 2021-10-01 09:00 UTC; computed duration unknown (vendor reported 11 minutes). Service recovered when locks expired; Doppler refactored the affected component to avoid locking. | Major incident |