# Mezmo: Critical incident: Log lines not ingested

**Summary:** Mezmo reported a critical log-ingestion outage across all sources from 2022-06-02 21:44 UTC to 2022-06-02 22:05 UTC (21 min). Newly submitted logs were never ingested, making them unavailable for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving. Ingestion resumed and all services were fully operational. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2022-06-02 21:44 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/fb84z2gjgwr6](https://stspg.io/fb84z2gjgwr6)
- [Company: Mezmo](https://rewindex.com/company/mezmo.md)
- Versions: before (none: first archived) → after incident\_version:282/z6dd84zdk8hv@c67a62b474af
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Log lines not ingested
```

**2. Added: Impact**

```diff
+ critical
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2022-06-02 21:44 UTC
```

**5. Added: Resolved time**

```diff
+ 2022-06-02 22:05 UTC
```

**6. Added: Affected components**

```diff
+ Log Ingestion (Agent/REST API/Code Libraries), Log Ingestion (Heroku), Log Ingestion (Syslog)
```

**7. Added: Update** (identified) · 2022-06-02 21:44 UTC

```diff
+ Ingestion of log lines has halted from all sources.  We are taking remedial action.
```

**8. Added: Update** (resolved) · 2022-06-02 22:05 UTC

```diff
+ Ingestion has resumed.  All services are fully operational.
```

**9. Added: Update** (postmortem) · 2022-06-03 22:49 UTC

```diff
+ **Dates:**   Start Time:	Thursday, June 2, 2022, at 20:25 UTC   End Time: 	Thursday, June 2, 2022, at 20:50 UTC   Duration:	00:25    **What happened:** The ingestion of logs was halted for about 25 minutes. During that time, newly submitted logs were never ingested and therefore not available for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving.    **Why it happened:** We manually reverted our ingester service to an older version \(to solve a minor problem unrelated to this incident\).  During the procedure, the version of the container was reverted, but not the container’s configuration.  Because of this versioning mismatch, logs from the ingester stopped being accepted by a downstream service \(the “buzzsaw broker”\).  The ingester is currently not designed to confirm logs are accepted by downstream services; therefore it returned http 200 messages to our customer’s agents, indicating logs had been successfully received.  At this point the agent discarded any locally cached log files.  Consequently, all log lines sent during the incident \(25 minutes\) were never ingested.    **How we fixed it:** We reverted the container’s configuration correctly, so it matched the version of the container itself.  Ingestion began working normally again.    **What we are doing to prevent it from happening again:** We will review and update our runbooks for reverting services to earlier versions to prevent similar mistakes.  We also plan to automate the reversion process. We will add internal confirmations to the ingester so it is always certain log lines were received by downstream services.  This will prevent the ingester from sending erroneous 200 messages back to the agent, should the ingester be unable to pass log lines downstream.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2022-06-02 21:44 UTC | Critical incident: Log lines not ingested (this change) | outage |

All Mezmo changes: [timeline](https://rewindex.com/company/mezmo/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i851075ac85705f · Markdown: https://rewindex.com/change/i851075ac85705f.md · JSON: https://rewindex.com/api/v1/changes/i851075ac85705f · Generated 2026-10-18T01:30:26.025Z