Changes / Mezmo / Alerts not triggering
Mezmo: Critical incident: Alerts not triggering
Summary
Mezmo’s critical Alerting outage ran from 2022-04-05 18:23 UTC to 2022-04-05 18:32 UTC (9 min); alerts failed for all accounts before service resumed. The postmortem mentions a parser service restart and says most alerts with trigger dates more than 15 minutes in the past were discarded.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Alerts not triggering
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2022-04-05 18:23 UTC
Before: not present
After: 2022-04-05 18:32 UTC
Before: not present
After: Alerting
Before: not present
After: Currently, alerts of all types are not triggering. We are taking remedial action.
Before: not present
After: Alerts of all types have resumed. Alerts are fully operational.
Before: not present
After: **Dates:**
Start Time: Tuesday, April 5, 2022, at 13:20:00 UTC
End Time: Tuesday, April 5, 2022, at 18:20:00 UTC
Duration: 5:00:00
**What happened:**
Alerting was halted for all accounts for the entire duration of the incident. Most alerts – any whose trigger date was more than 15 minutes in the past – were discarded.
**Why it happened:**
We restarted our parser service, for reasons unrelated to this incident. Any restart of the parser service should be followed by a restart of the alerting service. This second step was overlooked and didn’t happen. Subsequently, all alerts stopped triggering.
The need to restart alerting after a restart of the parser is already documented and well-known to our infrastructure team. However, the restart of the parser was performed by a team less familiar with the correct procedure.
**How we fixed it:**
We manually restarted the alerting service, which then returned to normal operation.
**What we are doing to prevent it from happening again:**
The proper documented restart procedure has been discussed with all teams allowed to restart services.
We will add monitoring of our alerting service and automated notifications so we learn more quickly of any similar incidents in the future.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Alerts not triggering (this change) Mezmo’s critical Alerting outage ran from 2022-04-05 18:23 UTC to 2022-04-05 18:32 UTC (9 min); alerts failed for all accounts before service resumed. The postmortem mentions a parser service restart and says most alerts with trigger dates more than 15 minutes in the past were discarded. | Major incident |