Changes / Close / Elevated Error Rate
Close: Major incident: Elevated Error Rate
Summary
Close reported a major incident affecting API and Application UI, with elevated error rates and possible request timeouts. It started at 2023-09-05 16:08 UTC and was resolved at 2023-09-05 17:04 UTC; the recorded duration was 55 min. A fix was deployed by 16:17 UTC, restoring normal application operation.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Elevated Error Rate
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-09-05 16:08 UTC
Before: not present
After: 2023-09-05 17:04 UTC
Before: not present
After: API, Application UI
Before: not present
After: Customers may be experiencing elevated error rates. The root cause has been identified. A fix is being deployed.
Before: not present
After: The fix has been deployed. The Close application is now functioning normally. Close Engineering is continuing to monitor application performance.
Before: not present
After: This incident has been resolved.
Before: not present
After: Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.
## Impact
Close users experienced elevated error rates from 16:00 until 16:16 UTC on September 5, 2023. During this time requests may have presented an error or timed out.
## Root Cause and Resolution
At 16:00 UTC on September 5, 2023 a version of the Close app with a major performance issue was deployed to production. This resulted in degraded performance for all customers from 16:00 until 16:16 UTC. Close Engineering deployed a new version of the the Close app that did not have the performance issue at 16:16 UTC. Application performance then returned to normal.
Close Engineering is working on a new set of pre-deployment tests that will prevent similar performance issues from being introduced in the future.
## Timeline
* 2023-09-05 16:00 - A new version of the Close app is published.
* 2023-09-05 16:03 - Close Engineering becomes aware of degraded performance of the Close application.
* 2023-09-05 16:05 - Close Engineering reverts to the last working version of the Close application.
* 2023-09-05 16:16 - The working version of the Close application is deployed. Performance recovers.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Elevated Error Rate (this change) Close reported a major incident affecting API and Application UI, with elevated error rates and possible request timeouts. It started at 2023-09-05 16:08 UTC and was resolved at 2023-09-05 17:04 UTC; the recorded duration was 55 min. A fix was deployed by 16:17 UTC, restoring normal application operation. | Major incident |