Rewindex

ChangesLoopsTransactional email, webhooks, SMTP interruptions

Loops: Major incident: Transactional email, webhooks, SMTP interruptions

Major incidentmajor (vendor)Changed Detected stspg.io (opens in new tab)
Summary

Summary pending. The diff below shows exactly what changed.

Incident page: the full timeline of updates

Part of an outage event

AWS us-east-1 outage, Oct 20, 2025 · 141 vendors, 181 incidents

Loops’s incident blames AWS as the cause. (hop 1, posted +1 h 20 min vs AWS’s own post)

“…ansactional email, webhooks, SMTP interruptions On October 20 2025, Loops was impacted by AWS’s outage in the US-EAST-1 region. We observed a variety of misbehaviors during that time, some more seriou…”

See the cascade chart and every vendor →

Detected automatically from incidents that started together and named the same provider. A link means one vendor’s incident named another, not a confirmed cause.

What changed

Struck in red is what Loops removed; blue is what they added. Muted text is unchanged context.

AddedTitle#1
Before: not present
After: Transactional email, webhooks, SMTP interruptions
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: postmortem
AddedStart time#4
Before: not present
After: 2025-10-20 08:30 UTC
AddedResolved time#5
Before: not present
After: 2025-10-20 10:47 UTC
AddedAffected components#6
Before: not present
After: Email Sending - Transactional, Transactional
AddedUpdatemonitoring#7
Before: not present
After: An issue in AWS is causing transactional emails to fail.
AddedUpdatemonitoring#8
Before: not present
After: We are continuing to monitor for any further issues.
AddedUpdatemonitoring#9
Before: not present
After: We are continuing to monitor for any further issues.
AddedUpdateresolved#10
Before: not present
After: This incident has been resolved.
AddedUpdatepostmortem#11
Before: not present
After: On October 20 2025, Loops was impacted by AWS’s outage in the US-EAST-1 region. We observed a variety of misbehaviors during that time, some more serious than others. Overall, the impact to Loops fell within the wider window provided by AWS \(06:49 UTC to 22:53 UTC\). During the outage, we experienced temporary but notable impact to, * Transactional email sending. * Webhook delivery. * Incoming request routing to our SMTP service. **Transactional email** Between 7:02 UTC and 9:18 UTC, transactional email sending was impacted. During this window the Loops API may have accepted transactional requests but failed to send them. Impacted messages were not resent later as we do not store the raw payloads. Timeline: * 7:02 UTC: We began experiencing elevated error rates related to transactional email sending, manifesting as increasing rates of HTTP 500 responses from AWS APIs. * 9:18 UTC: AWS API error rates subside. Transactional sending returns to normal. **Webhooks** Between 9:30 UTC and 10:31 UTC webhook delivery was impacted due to the inability to provision sufficient EC2 capacity. Timeline: * 9:30 UTC: Error rates between the Loops API and our backend webhooks service rise. * 10:01 UTC: The issue is identified as insufficient capacity for the webhooks service due to EC2 provisioning scarcity. * 10:31 UTC: Error rates subside as we change our autoscaling configurations to accept a wider range of EC2 instance types and sizes. **SMTP availability** Between approximately 19:33 UTC and 21:35 UTC an incorrect DNS entry was returned in the results for the network load balancer \(NLB\) used by our SMTP service. During this window, one of three IP addresses returned by DNS responses were unable to accept connections. Attempts to connect to the incorrect IP address manifested as timeouts. Timeline: * ~19:33 UTC: An IP address returned NLB DNS requests stops accepting connections. DNS requests that select the impacted IP address fail to connect and time out. * ~21:35 UTC: The DNS for the NLB corrects itself and begins returning correct responses.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

Monday, October 20, 2025

1 incident change

All Loops changesLoops profileMarkdownJSON