Companies / Front / incidents

Front: Increased error rate sending and receiving messages on email-based channels

Impact: majorStatus: resolvedStarted Resolved (6.4 h)8 updatesSource: stspg.io (opens in new tab)
Summary

Front reported a major incident affecting Gmail, O365, and SMTP (non-Gmail / O365) message sending and receiving from 2024-09-27 20:25 UTC to 2024-09-28 02:51 UTC (6 h 26 min). A manually deployed fix circumvented a Docker outage; delivery recovered with residual backlog delays. us-west-2 and eu-west-1 recovered before us-west-1.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: SMTP (non-Gmail / O365)GmailO365

Timeline of updates

  1. investigating

    We are currently investigating this issue.

  2. investigating

    We are continuing to investigate this issue.

  3. identified

    We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.

  4. identified

    We are validating the fix in our staging environment.

  5. identified

    We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.

  6. monitoring

    The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.

  7. monitoring

    Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.

  8. resolved

    This incident has been resolved.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Front profile · Recent Front incidents · Markdown · JSON