# Mezmo: Major incident: Email Alerts Are Not Being Sent

**Summary:** Mezmo reported a major email-alerting outage starting 2021-10-06 21:29 UTC, lasting 1 h 5 min and resolving at 2021-10-06 22:35 UTC; Slack and webhook alerts remained functional. The postmortem attributed partial email-notification disruption to exceeding its email provider’s daily limit of 250,000 messages; email alerting was restored. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2021-10-06 21:29 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/8wqg81wb0y9t](https://stspg.io/8wqg81wb0y9t)
- [Company: Mezmo](https://rewindex.com/company/mezmo.md)
- Versions: before (none: first archived) → after incident\_version:282/wx1p648vyh1x@b1cc1960b48c
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Email Alerts Are Not Being Sent
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2021-10-06 21:29 UTC
```

**5. Added: Resolved time**

```diff
+ 2021-10-06 22:35 UTC
```

**6. Added: Affected components**

```diff
+ Alerting
```

**7. Added: Update** (investigating) · 2021-10-06 21:29 UTC

```diff
+ Our email alerting feature is not working at the moment and customers are not receiving alerts by email. Other types of alerts, such as Slack and webhook, are still working. We are investigating.
```

**8. Added: Update** (resolved) · 2021-10-06 22:35 UTC

```diff
+ Our email alerting feature has been restored to normal operation. All services are fully functional.
```

**9. Added: Update** (postmortem) · 2021-10-13 21:01 UTC

```diff
+ Start Time:		Wednesday, October 6, 2021, at 17:30:06 UTC   End Time: 		Wednesday, October 6, 2021, at 21:30:27 UTC   Duration:		         4:00:21 ## What happened: Email notifications from alerts were partially halted for about 4 hours.  Notifications sent by Slack and Webhooks were not affected. ## Why it happened: Our email service provider’s daily limit of 250,000 email messages was exceeded.  All email message notifications from triggered alerts bounced and could not be resent. Further investigation revealed that a HackerOne Security Analyst looking for flaws in our service to report to us had made our system send 450,000 emails.  This was accomplished by manually adding an array of many email addresses into the “Change Owner” request field within our Web UI. Adding multiple email addresses in that field is not permitted through normal usage of the Web UI. The Security Analyst intercepted the http request sent when the form was submitted and manually inserted a json list in the field, effectively sending an array of email addresses rather than a string.  LogDNA had no server-side \(e.g. backend\) validation to ensure only a string could be accepted. ## How we fixed it: We took remedial action by contacting our email provider, who temporarily increased our daily email sending limit to 625,000 messages.  This allowed email notifications from alerts to resume. We then added server-side validation for the “Change Owner” field in our Web UI so that only strings are accepted, even if the request is manually intercepted and an array of email addresses is added. ## What we are doing to prevent it from happening again: We will audit our Web UI to find all places where multiple email inputs can be added.  We’ll then add server-side validation, so only strings are accepted. We’ll emphasize to our HackerOne Security Analysts that they should not take potentially damaging actions as they proactively search for vulnerabilities in our service.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2021-10-06 21:29 UTC | Major incident: Email Alerts Are Not Being Sent (this change) | outage |

All Mezmo changes: [timeline](https://rewindex.com/company/mezmo/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/ic9135ae612e997 · Markdown: https://rewindex.com/change/ic9135ae612e997.md · JSON: https://rewindex.com/api/v1/changes/ic9135ae612e997 · Generated 2026-10-18T03:29:40.548Z