Changes / Iterable / Event-Triggered Journeys Delays Ingesting New Users
Iterable: Major incident: Event-Triggered Journeys Delays Ingesting New Users
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Event-Triggered Journeys Delays Ingesting New Users
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-12-31 14:29 UTC
Before: not present
After: 2024-12-31 18:25 UTC
Before: not present
After: Global Web Application
Before: not present
After: Summary:
Event-Triggered Journeys Delays Ingesting New Users. The issue originated from errors in the workflow-entrance-trigger pods, causing a significant backlog in processing. There is no impact to Scheduled Journeys and API Triggered Journeys .
Actions Taken
The workflow-entrance-trigger service was updated to the latest version, and additional pods were scaled up to process the backlog faster. The deployment resolved the issue, and error rates dropped significantly.
Current Status
The errors we were experiencing have been fixed since 5AM PST, now we're just monitoring the backlog as it drains. For 99% of clients, the backlog has drained completely, there are a few stragglers with small backlogs
Next Steps
Engineers will continue monitoring error rates and ensure the backlog clears entirely. Follow-up tasks include setting up error rate monitoring and addressing journey-specific issues to prevent recurrence.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: This issue is now resolved and the backlog is completely drained. Event trigger journey's as of 7:25 AM PST are back to normal and processing as expected.
If you still have any further questions please reach out to support@iterable.com.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Event-Triggered Journeys Delays Ingesting New Users (this change) | Major incident |