rewindex

CompaniesCloseIncidents

Close: Degraded Dialer Performance

MinorStatus: postmortemStarted Resolved (25 min)4 updatesstspg.io (opens in new tab)

Affected components: DialerPhone - Dialer

Timeline of updates

  1. investigating

    We've become aware of degraded performance of our Dialer service. We are investigating the issue. Updates will be posted as they become available.

  2. monitoring

    We are continuing to investigate the cause of the degraded performance of our Dialer system. Our Dialer system is now functioning normally. We are monitoring performance.

  3. resolved

    This incident has been resolved.

  4. postmortem

    Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring.

    Impact

    Dialer functionality was impaired for 58 minutes from 15:20 UTC to 16:18 UTC on March 10th 2025. During this time the Dialer feature could get stuck in “connecting” state.

    Root Cause and Resolution

    The issue was triggered at 15:20 UTC by a service rebalance that caused a number of client connections to close simultaneously. When these clients attempted to reconnect, the sudden spike in traffic that occurred in peak traffic conditions exceeded system limits, leading to service disruptions.

    Our team quickly identified the cause and worked to stabilize the system. We restored normal operations by 16:18 UTC.

    To prevent similar incidents in the future, we are reviewing system thresholds and improving our ability to handle sudden increases in demand.

    Timeline

    • 15:20 UTC - a service rebalance occurs, starting a wave of new connections being established
    • 15:21 UTC - a portion of requests starts getting dropped due to rate limits
    • 15:28 UTC - alerts trigger and our response team began identifying the root cause
    • 15:36 UTC - the rate of dropped requests subsides, but then increases again soon due to a wave-like pattern of retries
    • 16:18 UTC - final wave of increased errors finishes and situation returns to normal operational levels

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Cite this

Status incident (vendor impact label: minor) posted by Close, started 2025-03-10 16:02 UTC, resolved after 25 min, 4 update(s).

Title
Degraded Dialer Performance
Vendor
Close
Date
URL
https://rewindex.com/company/close/incidents/x7sfct92xpkx

Close profileRecent Close incidentsAll outages on March 10, 2025Close in March 2025MarkdownJSON