Companies / Iterable / incidents

Iterable: Ingestion Delay On Subset Of Clusters

Affected components: Cluster 16 - Workflow ProcessingJourney ProcessingCluster 14 - User DeletionsUser DeletionsCluster 14 - Email SendsEmail SendsCluster 22 - List UploadsList UploadsCluster 22 - Workflow ProcessingJourney ProcessingCluster 115 - Workflow ProcessingJourney ProcessingCluster 125 - Workflow ProcessingJourney ProcessingCluster 115 - List UpdatesList UpdatesCluster 22 - User DeletionsUser DeletionsCluster 14 - List UploadsList UploadsCluster 125 - User UpdatesUser UpdatesCluster 125 - List UpdatesList UpdatesCluster 14 - Workflow ProcessingJourney ProcessingCluster 25 - List UpdatesList UpdatesCluster 125 - User DeletionsUser DeletionsCluster 14 - User UpdatesUser UpdatesCluster 115 - User UpdatesUser UpdatesCluster 125 - Email SendsEmail SendsCluster 25 - Workflow ProcessingJourney ProcessingCluster 16 - User DeletionsUser DeletionsCluster 25 - Email SendsEmail SendsCluster 115 - Email SendsEmail SendsCluster 115 - User DeletionsUser DeletionsCluster 16 - List UploadsList UploadsCluster 22 - User UpdatesUser UpdatesCluster 22 - Email SendsEmail SendsCluster 25 - User UpdatesUser UpdatesCluster 16 - Email SendsEmail SendsCluster 25 - User DeletionsUser DeletionsCluster 16 - User UpdatesUser Updates

Timeline of updates

1. investigating2024-11-12 17:18 UTC

   Starting at around 6:30 am PT, Iterable Engineers were notified that lag on our ingestion topics was growing on a subset of clusters. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:00 an PT or Sooner.

2. monitoring2024-11-12 17:52 UTC

   The engineering team has identified the cause and remediated the delays on these subset of clusters. Most impacted ingestion topics have recovered and customers should start to see data being ingested normally again. We are still working through some backlog, so the team will continue to monitor to ensure that these delays don't recur. Next Update at 11:00 am PT or sooner?

3. resolved2024-11-12 19:24 UTC

   Ingestion has returned to normal for all impacted clusters and this issue is now resolved.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Iterable profile · Recent Iterable incidents · Markdown · JSON