rewindex

CompaniesCloseIncidents

Close: Background Processing Delayed

MinorStatus: postmortemStarted Resolved (54 min)7 updatesstspg.io (opens in new tab)

Affected components: Application UIAPI

Timeline of updates

  1. investigating

    We are currently investigating this issue.

  2. identified

    The issue has been identified and a fix is being implemented.

  3. identified

    We are continuing to work on a fix for this issue.

  4. monitoring

    A fix has been implemented and we are monitoring the results.

  5. monitoring

    A fix has been implemented. The Close application and API are fully functional. We expect full recovery of background task processing performance in the next 15 minutes.

  6. resolved

    This incident has been resolved.

  7. postmortem

    Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

    Impact

    Our processing of background tasks was delayed between 13:30 and 15:30 UTC on September 15, 2023. Some customers may have noticed the Close Application taking longer than normal to sync data from third parties like Zoom or Calendly. Inbox notifications may have been delayed. Some Power Dialer sessions may also have been impacted. No data was lost during this incident.

    Root Cause and Resolution

    At 13:25 UTC on September 15, 2023 a new version of the Close Application was deployed to production. This change introduced a bug into our background task processing system that prevented it from processing new work. By 14:35 Close Engineering deployed a a fix and performance began to recover. The Close Application completely recovered by 15:30 UTC.

    To prevent similar issues from happening in the future, Close Engineering will upgrade our background task processing system to better isolate workloads that could potentially break. 

    Timeline

    • 13:25 UTC - A new version of the Close app is published.
    • 13:39 UTC - Close Engineering becomes aware of a problem with background task processing and takes steps to remediate.
    • 14:35 UTC - Close Engineering deploys a fixed version of the Close application. Performance begins to recover.
    • 15:30 UTC - The Close Application is fully recovered and performing normally.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Cite this

Status incident (vendor impact label: minor) posted by Close, started 2023-09-15 14:37 UTC, resolved after 54 min, 7 update(s).

Title
Background Processing Delayed
Vendor
Close
Date
URL
https://rewindex.com/company/close/incidents/g78ndttj4lht

Close profileRecent Close incidentsAll outages on September 15, 2023Close in September 2023MarkdownJSON