Changes / Mezmo / Ingestion of new logs — for Syslog only - is intermittently failing
Mezmo: Major incident: Ingestion of new logs — for Syslog only - is intermittently failing
Summary
Mezmo reported a major degradation: new-log ingestion through its Syslog endpoint intermittently failed from 2022-02-18 01:41 UTC to 2022-02-18 02:15 UTC (33 min). The incident was resolved; the postmortem attributed the failures to a code change in Syslog Forwarder, which handles Syslog ingestion.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 75%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Ingestion of new logs — for Syslog only - is intermittently failing
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2022-02-18 01:41 UTC
Before: not present
After: 2022-02-18 02:15 UTC
Before: not present
After: Log Ingestion (Syslog)
Before: not present
After: Ingestion of new logs to our Syslog endpoint is intermittently failing. We are investigating.
Before: not present
After: This incident has been resolved. If your team is still unable to send logs via syslog, please let us know at support@logdna.com
Before: not present
After: **Dates:**
Start Time: Wednesday, February 17, 2022, at 20:56 UTC
End Time: Thursday, February 18, 2022, at 02:15 UTC
Duration: 5:19:00
**What happened:**
The ingestion of new logs to our Syslog endpoint was intermittently failing.
**Why it happened:**
We made a code change to the area of our service \(Syslog Forwarder\) that handles the ingestion of logs sent by Syslog and inadvertently changed how memory is managed. Routine memory garbage collection stopped and memory usage increased on the pods that accept newly submitted log lines over Syslog. Eventually, the increase in memory caused the pods to crash. Any log lines held on those pods were lost and never ingested.
**How we fixed it:**
We reverted to the previous version of the Syslog Forwarder service. This stopped the pods from crashing.
We then resolved the memory management issue in our code. The new, fixed version was released to production shortly thereafter and performed as expected.
**What we are doing to prevent it from happening again:**
We have added regression tests to the Syslog Forwarder service to prevent a similar mistake in the future.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Ingestion of new logs — for Syslog only - is intermittently failing (this change) Mezmo reported a major degradation: new-log ingestion through its Syslog endpoint intermittently failed from 2022-02-18 01:41 UTC to 2022-02-18 02:15 UTC (33 min). The incident was resolved; the postmortem attributed the failures to a code change in Syslog Forwarder, which handles Syslog ingestion. | Major incident |