rewindex

CompaniesFullStoryIncidents

FullStory: Fullstory APIs Unavailable

No impact listedStatus: postmortemStarted Resolved (-184 min)2 updatesstspg.io (opens in new tab)

Timeline of updates

  1. resolved

    From October 31, 2024 at 2:38PM ET through October 31, 2024 at 4:07PM ET, the Fullstory APIs were unintentionally unavailable. The majority of requests to Fullstory APIs would have returned an error. Fullstory customers that attempted to send an API request during the outage window will need to process these requests again.

    If you believe your account was impacted or have any questions, please reach out to support@fullstory.com.

  2. postmortem

    On October 31, 2024, our API service experienced a disruption that resulted in a significant number of dropped requests and errors. The incident lasted for approximately 1.5 hours and impacted all users of our public server API.

    Customer Impact

    Most public server API requests sent during the incident time frame (18:37:29 UTC to 20:12:39 UTC) were unsuccessful and should be re-submitted. We acknowledge the inconvenience and disruption this caused our customers and sincerely apologize for any impact on your operations.

    Root Cause

    The root cause of the incident was a decrease in the capacity of our API service due to an unsuccessful deployment. This decrease in capacity, combined with higher than usual request volume, led to the service being unable to handle the incoming traffic.

    Resolution

    Upon discovery of the issue, our on-call operations team was immediately notified and responded swiftly. The team's actions included re-executing the failed deployment to restore normal API service functionality. 

    Process Changes and Prevention

    Actions Taken:

    • We re-executed the unsuccessful deployment.
    • We permanently scaled up the capacity of our API service to handle the increased traffic.
    • We are implementing additional monitoring and alerting to detect and prevent similar issues in the future.

    Ongoing Improvements:

    • We are reviewing our deployment processes to minimize the risk of similar incidents.
    • We are investigating ways to improve the resilience of our API service to handle unexpected spikes in traffic.

    ‌

    We deeply regret this incident and invite any Fullstory customer who was materially affected to contact [support@fullstory.com](mailto:support@fullstory.com). We stand by ready to fully address all of your concerns.

Archive proof

Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert. That proves this record existed by then and has not been changed since, by anyone, Rewindex included.

How the archive is checked, and what it cannot prove.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab).

Cite this

Status incident (vendor impact label: none) posted by FullStory, started 2024-10-31 21:33 UTC, resolved after -184 min, 2 update(s).

Title
Fullstory APIs Unavailable
Vendor
FullStory
Date
URL
https://rewindex.com/company/fullstory/incidents/b7fmzq9f9p64

FullStory profileRecent FullStory incidentsAll outages on October 31, 2024FullStory in October 2024MarkdownJSON