rewindex

CompaniesCloseIncidents

Close: Search Indexer Delays

MinorStatus: postmortemStarted Resolved (46 min)5 updatesstspg.io (opens in new tab)

Affected components: QueryingSearch - QueryingApplication UIIndexingSearch - IndexingAPI

Timeline of updates

  1. investigating

    We are investigating an issue where our Search Indexer's ingestion of new events is delayed. This may result in the latest changes not being reflected on many pages across the application – Leads Search, Contacts Search, Inbox, Opportunities, Reporting, etc.

  2. identified

    We have identified a problem with one of our search databases and have initiated a failover.

  3. monitoring

    The failover succeeded. The Indexer is still catching up with the backlog of events. We are monitoring the situation and estimate that everything will be back to normal within 3-5 minutes.

  4. resolved

    The issue is resolved now.

  5. postmortem

    Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. 

    Impact

    For some clients changes performed in Close took longer than usual to be reflected in our search functionality on November 4th from 8:06 to 9:31 UTC.

    Root Cause and Resolution

    We are in the process of moving data to a new search infrastructure. Due to encountering a number of unexpectedly large objects during this process the indexing of new changes got delayed. We adjusted the indexing process to truncate the overly large objects to avoid encountering this in the future.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Cite this

Status incident (vendor impact label: minor) posted by Close, started 2024-11-04 08:45 UTC, resolved after 46 min, 5 update(s).

Title
Search Indexer Delays
Vendor
Close
Date
URL
https://rewindex.com/company/close/incidents/f3xzqb5n2yxl

Close profileRecent Close incidentsAll outages on November 4, 2024Close in November 2024MarkdownJSON