Changes / Close / Degraded Performance
Close: Major incident: Degraded Performance
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Degraded Performance
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-02-11 22:31 UTC
Before: not present
After: 2025-02-11 23:34 UTC
Before: not present
After: API, Application UI, Calendar - Syncing, Email - Syncing, Syncing
Before: not present
After: We have become aware of degraded performance of the Close system. We are actively triaging the issue. Updates will be posted as they become available.
Before: not present
After: The issue has been identified. A fix is being rolled out to our production system.
Before: not present
After: A fix has been implemented. We are monitoring the results.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: Our App UI and API have returned to normal performance. Our email and calendar syncing systems are operational but delayed. We are continuing to monitor for further issues.
Before: not present
After: This incident has been resolved.
Before: not present
After: Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.
## Impact
Beginning 22:00 UTC until 23:03 UTC on Tuesday February 11, 2025 users of the Close App experienced significantly degraded performance and an elevated error rate. This included slow page loads, long API response times, delayed email syncing, and delayed calendar syncing for all customers.
## Root Cause and Resolution
At 22:00 UTC on Tuesday February 11, 2025 a change was deployed to the production environment of the Close system. This change introduced a poorly optimized database query. This caused our primary back end database to become overloaded and unstable. Close Engineering identified and removed the offending database query by 22:47 UTC. Performance began to gradually return to baseline, fully recovering by 23:03 UTC.
## Timeline
* 22:00 UTC - A change is deployed to production introducing a poorly optimized database query
* 22:01 UTC - Load on our primary back end database begins to climb
* 22:06 UTC - Load on our primary back end database breaches critical thresholds alerting Close Engineering
* 22:17 UTC - Close Engineering isolates the poorly optimized database query and begins to remove it from production
* 22:47 UTC - Removal of the poorly optimized database query completes
* 23:03 UTC - Close system performance returns to baseline
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Degraded Performance (this change) | Major incident |