Changes / Mezmo / Ingestion of new logs — for Syslog only - Partial Outage
Mezmo: Major incident: Ingestion of new logs — for Syslog only - Partial Outage
Summary
Mezmo reported major, Syslog-only log ingestion failures from 2022-02-24 22:45 UTC to resolution at 2022-02-24 23:43 UTC (57 min computed). New logs intermittently failed to ingest. The postmortem gives an earlier start—Thursday, February 18, 2022, at 00:10 UTC—and mentions the newly introduced Syslog Forwarder service.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 60%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Ingestion of new logs — for Syslog only - Partial Outage
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2022-02-24 22:45 UTC
Before: not present
After: 2022-02-24 23:43 UTC
Before: not present
After: Log Ingestion (Syslog)
Before: not present
After: New logs — from Syslog only -- are intermittently not being ingested by our service. We are working to restore this functionality as soon as possible.
Before: not present
After: This incident has been resolved. Please reach out to us at support@logdna.com with any additional questions.
Before: not present
After: **Dates:**
Start Time: Thursday, February 18, 2022, at 00:10 UTC
End Time: Thursday, February 24, 2022, at 23:43 UTC
Duration: 167:33:00
**What happened:**
The ingestion of new logs to our Syslog endpoint was intermittently failing.
**Why it happened:**
We recently introduced a new service \(Syslog Forwarder\) to handle the ingestion of logs sent over Syslog. As the name implies, it forwards logs to downstream services. It was designed to send all logs submitted for each account to a single port opened on the downstream services. No load balancing was implemented in our original design, which performed well in our advance testing.
Once put into production, however, it became apparent that some customer accounts submit logs at a volume higher than the downstream services could process. When this happened, logs lines were buffered in memory by the Syslog Forwarder. Memory increased until the pods crashed. Any log lines held on those pods were lost and never ingested.
**How we fixed it:**
We improved the design of the Syslog Forwarder by adding a pool of connections to the downstream services. In effect, we added traffic shaping to the Syslog Forwarder.
**What we are doing to prevent it from happening again:**
The new architecture has been incorporated and proven resilient in production. No further work is needed to prevent this kind of incident from happening again.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Ingestion of new logs — for Syslog only - Partial Outage (this change) Mezmo reported major, Syslog-only log ingestion failures from 2022-02-24 22:45 UTC to resolution at 2022-02-24 23:43 UTC (57 min computed). New logs intermittently failed to ingest. The postmortem gives an earlier start—Thursday, February 18, 2022, at 00:10 UTC—and mentions the newly introduced Syslog Forwarder service. | Major incident |