# Front: Major incident: Increased error rate sending and receiving messages on email-based channels

**Summary:** Front reported a major incident affecting Gmail, O365, and SMTP (non-Gmail / O365) message sending and receiving from 2024-09-27 20:25 UTC to 2024-09-28 02:51 UTC (6 h 26 min). A manually deployed fix circumvented a Docker outage; delivery recovered with residual backlog delays. us-west-2 and eu-west-1 recovered before us-west-1. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2024-09-27 20:25 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/7znnyb34f8fn](https://stspg.io/7znnyb34f8fn)
- [Company: Front](https://rewindex.com/company/front.md)
- Versions: before (none: first archived) → after incident\_version:328/2hhhx61h8cnh@ba33cd3a41cf
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Increased error rate sending and receiving messages on email-based channels
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ resolved
```

**4. Added: Start time**

```diff
+ 2024-09-27 20:25 UTC
```

**5. Added: Resolved time**

```diff
+ 2024-09-28 02:51 UTC
```

**6. Added: Affected components**

```diff
+ Gmail, O365, SMTP (non-Gmail / O365)
```

**7. Added: Update** (investigating) · 2024-09-27 20:25 UTC

```diff
+ We are currently investigating this issue.
```

**8. Added: Update** (investigating) · 2024-09-27 20:25 UTC

```diff
+ We are continuing to investigate this issue.
```

**9. Added: Update** (identified) · 2024-09-27 20:38 UTC

```diff
+ We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.
```

**10. Added: Update** (identified) · 2024-09-27 21:02 UTC

```diff
+ We are validating the fix in our staging environment.
```

**11. Added: Update** (identified) · 2024-09-27 21:27 UTC

```diff
+ We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.
```

**12. Added: Update** (monitoring) · 2024-09-27 21:47 UTC

```diff
+ The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.
```

**13. Added: Update** (monitoring) · 2024-09-28 02:21 UTC

```diff
+ Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.
```

**14. Added: Update** (resolved) · 2024-09-28 02:51 UTC

```diff
+ This incident has been resolved.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2024-09-27 20:25 UTC | Major incident: Increased error rate sending and receiving messages on email-based channels (this change) | outage |

All Front changes: [timeline](https://rewindex.com/company/front/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ia93743612e3df5 · Markdown: https://rewindex.com/change/ia93743612e3df5.md · JSON: https://rewindex.com/api/v1/changes/ia93743612e3df5 · Generated 2026-10-18T02:31:44.084Z