Search

Companies / Iterable / incidents

Iterable: Queries to data stored on C120 cluster are failing

Impact: majorStatus: resolvedStarted Resolved (40 h)8 updatesSource: stspg.io (opens in new tab)
Summary

Iterable reported a major C120 outage starting 2025-04-23 06:50 UTC, lasting 40 h 26 min and resolved at 2025-04-24 23:17 UTC. A cluster formation problem caused query failures, campaign impacts and ingestion lag; users triggering journeys via custom events, system events or profile updates did not enter journeys. Engineering restored data and cut over to Cluster 141.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Cluster 120 - List UpdatesList UpdatesCluster 120 - Email SendsEmail SendsCluster 120 - User UpdatesUser UpdatesCluster 120 - Push SendsPush SendsCluster 120 - User DeletionsUser DeletionsCluster 120 - Workflow ProcessingJourney ProcessingCluster 120 - SMS SendsSMS Sends

Timeline of updates

  1. investigating

    All queries to Cluster C120 are failing as there was a problem in cluster formation. The problem started around 4/22/2025 19:10 pm PDT. Any campaign sent during this time would have been impacted. Ingestion lag is currently building up. The team is restoring the data into an alternate cluster to bring back all organizations into a working state. This can take several hours. The ingestion lag will deplete after the organizations are restored. No data should be lost. Next update at 4/23/2025 6 am PDT.

  2. investigating

    We are continuing to investigate this issue.

  3. investigating

    The team has finished restoring data from C120 to a new cluster. Engineering team is working on migrating the in-flight data. This can take few hours, and we will update once we receive a time frame. The ingestion lag will deplete once the data for the organizations are restored. No data should be lost, however event triggered journeys, exports to Snowflake, and campaign sends will be delayed. Next update will be at 4/23/2025 9 am PDT.

  4. identified

    The team is still working on a cutover to the new cluster. Once the changes propagate to all of the services we should start to see the ingestion lag deplete. Once cutover is complete Blast campaigns should resume. Next update will be at 4/23/2025 10 am PDT.

  5. identified

    Iterable's engineering team is continuing their work on the cutover to the new cluster. They are working on finalizing details to make sure we minimize impact as much as possible. Next update will be at 4/23/2025 11 am PDT.

  6. monitoring

    At this point all services have up-to date changes. Iterable's engineering team have cutover to the new cluster. The impacted customers are in the process of being notified. We are actively monitoring. Next update will be at 4/23/2025 around 12:15 pm PDT.

  7. monitoring

    Engineering successfully completed the cluster cutover at 11:03 AM PDT. Ingestion for customers previously on Cluster 120 is now flowing normally through the new Cluster 141. We’ve also confirmed that campaign scheduling has resumed and is running successfully on Cluster 141. Next update will be provided by 3:00 PM PDT.

  8. resolved

    Engineering has confirmed that the cluster is performing as expected. No new issues have occurred since the last update. We also wanted to share that further investigation post-incident has uncovered that, during the impact window, users that were set to start journeys via custom events, system events, and user profile updates did not enter journeys. However, for journeys initiated by scheduled, API calls, and "other journey" actions, the system functioned as expected, and users progressed through their journeys. The incident has now been marked as resolved. If you have questions, please reach out to our Support team.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Iterable profile · Recent Iterable incidents · Markdown · JSON