Companies / Iterable / incidents
Iterable: Web Application Errors
Iterable reported a major incident involving API and web application errors, with delays to sends and journey processing. It started at 2024-11-24 18:16 UTC and was resolved at 2024-11-25 17:09 UTC, lasting 22 h 52 min; engineering applied mitigations and reported recovery in the web application and API endpoints.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Global Partner WebhooksGlobal Web ApplicationGlobal API SuccessGlobal System WebhooksGlobal Campaign Sends
Timeline of updates
- investigating
Beginning at 5 AM PST we were alerted to a spike in API errors and Web App errors. Our engineering team is investigating. Next update will be at 7:30 AM PST or sooner. If you have questions please reach out to support@iterable.com
- investigating
Iterable engineering has put in mitigations and is seeing recovery in the web application and API endpoints. We are continuing to work on remediating delays to sends and journey processing. Next update by 8:30 am PT
- investigating
We're seeing continued improvements in the web application and API performance. We are still working to resolve delays in sends and webhooks. Next update by 9:00 AM PT or sooner
- investigating
We have put in further mitigations to restore send speeds and are working through the send backlog. We are continuing to work on recovering webhooks. Next update by 9:30 AM PT or sooner
- investigating
The Engineering team is investigating web and API errors. Next update by 9:45 AM PT or sooner.
- investigating
The Engineering team is implementing mitigations to address the intermittent web and API errors as well as send and webhook delays. Next update by 10:15 AM PT or sooner.
- investigating
The Engineering team is continuing to investigate and implement mitigations to address DB issues causing web and API errors. Sends, journey processing, and webhooks will continue to see delays. Customers may also experience issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 10:45 AM PT or sooner.
- investigating
The Engineering team is continuing to investigate and implement mitigations to address DB issues causing web and API errors. Sends, journey processing, and webhooks will continue to see delays. Customers may also experience issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 11:15 AM PT or sooner.
- investigating
The Engineering team is continuing to investigate and implement mitigations to address DB issues. Web errors and journey latency are going down and send rate is going up. Customers may still experience degradation and issues with pages of the web app failing to fully load and an inability to switch projects. Next update by 11:45 AM PT or sooner.
- identified
The Engineering team is continuing to address DB issues. We are slowly bringing up the scaled down instances of various services. Web errors and journey latency are going down and send rate is going up. Customers may still experience issue with web app failing to fully load and an inability to switch projects. Next update by 12:15 PM PT or sooner.
- identified
The issue is now identified. The engineering team is slowly bringing up the scaled down instances of various services. Customers should begin seeing improvements in access to the web app, and message send rates are beginning to recover. All services except webhooks are scaled up now. We should continue to see improvements across the app as we catch up on the messaging backlog. Next update by 12:45 PM PT or sooner.
- identified
The issue is identified. The engineering team continues to scale up instances of various services. Customers should begin seeing improvements in access to the web app, and message send rates are beginning to recover. We should continue to see improvements across the app as we catch up on the messaging backlog. Next update by 1:15 PM PT or sooner.
- identified
The issue is identified. The engineering team continues to scale up instances of various services. Customers should continue seeing improvements in access to the web app, and message send rates continue to recover. We continue to see improvements across the app as we catch up on the messaging backlog. Next update by 2:15PM PT or sooner.
- monitoring
The engineering team has completed the instances scale up of various services. Access to the web app, and message send rates are back to normal processing. We continue to see improvements across the app as we catch up on the messaging backlog. Most systems are caught up with processing. Next update by 3:15PM PT or sooner.
- monitoring
Engineering team has confirmed that we continue to process at normal rates. Our processing of messaging backlog has caught up. Engineering is continuing to monitor our systems, if you are still seeing any impact please reach out to support@iterable.com
- monitoring
We are continuing to monitor for any further issues.
- monitoring
The Engineering team continues to monitor system performance. There have been no performance degradations. Next update by 12 pm PT or sooner. If you are still seeing any impact please reach out to support@iterable.com
- monitoring
This incident has been resolved.
- resolved
This incident has been resolved.
Rewindex changes for this incident
- Major incident: Web Application Errors 2024-11-24 18:16 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Iterable profile · Recent Iterable incidents · Markdown · JSON