Changes / Iterable / Ingestion Delay on Clusters 100 and Above
Iterable: Critical incident: Ingestion Delay on Clusters 100 and Above
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Ingestion Delay on Clusters 100 and Above
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2025-03-14 16:56 UTC
Before: not present
After: 2025-03-14 21:13 UTC
Before: not present
After: Global API Ingestion
Before: not present
After: Our engineering team has identified an issue causing ingestion delay for all Clusters 100 and above. We are deploying the fix right now. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:30 AM PDT or sooner.
Before: not present
After: The engineering team is still working on the fix, and customers on Clusters 100 and above will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 11:00 AM PDT or sooner.
Before: not present
After: Engineering has put in the fix. Some clusters have fully recovered, and we are actively working on the ones that have not. For those customers that are still impacted, you will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 12:30 PM PDT or sooner.
Before: not present
After: Engineering has finished pushing the fix. Customers on Clusters 116, 120, and 132 may still see an ingestion lag greater than 1 hour. All other Clusters have recovered. Data was never dropped. The next update is at 3 PM PDT or sooner.
Before: not present
After: Our engineering team has confirmed at 2:06 PDT that all ingestion should be back to normal and is stable. This incident is now resolved.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Ingestion Delay on Clusters 100 and Above (this change) | Major incident |