Changes / Front / Increased error rate sending and receiving messages on email-based channels
Front: Major incident: Increased error rate sending and receiving messages on email-based channels
Summary
Summary pending. The diff below shows exactly what changed.
What changed
Before: not present
After: Increased error rate sending and receiving messages on email-based channels
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-09-27 20:25 UTC
Before: not present
After: 2024-09-28 02:51 UTC
Before: not present
After: Gmail, O365, SMTP (non-Gmail / O365)
Before: not present
After: We are currently investigating this issue.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.
Before: not present
After: We are validating the fix in our staging environment.
Before: not present
After: We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.
Before: not present
After: The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.
Before: not present
After: Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Increased error rate sending and receiving messages on email-based channels (this change) | Major incident |