# Mezmo: Major incident: Searches are running slowly

**Summary:** Mezmo reported major Search degradation starting 2023-02-10 18:11 UTC, resolved at 2023-02-10 18:14 UTC (computed duration: 2 min). Searches were slow or returned no results; no data was lost and ingestion continued. The postmortem lists an earlier start of 16:45 UTC; normal search speeds were restored. _(machine-written, sol)_
- Kind: Major incident · Label: degradation (model (Jev))
- Changed: 2023-02-10 18:11 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/vhcf1kd1wcj3](https://stspg.io/vhcf1kd1wcj3)
- [Company: Mezmo](https://rewindex.com/company/mezmo.md)
- Versions: before (none: first archived) → after incident\_version:282/9jxtlgbtxhqk@18d299329685
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Searches are running slowly
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2023-02-10 18:11 UTC
```

**5. Added: Resolved time**

```diff
+ 2023-02-10 18:14 UTC
```

**6. Added: Affected components**

```diff
+ Search
```

**7. Added: Update** (identified) · 2023-02-10 18:11 UTC

```diff
+ Searches are running slowly. We have identified the cause and are implementing a fix. (Reference # 3018)
```

**8. Added: Update** (resolved) · 2023-02-10 18:14 UTC

```diff
+ Searches are running at normal speeds again.  All services are fully operational.
```

**9. Added: Update** (postmortem) · 2023-02-21 23:27 UTC

```diff
+ **Dates:**   Start Time: Friday, February 10, 2023 at 16:45 UTC   End Time:  Friday, February 10, 2023 at 18:14 UTC   Duration: 89 minutes    **What happened:** Searches returned results slowly or not at all.   No data was lost and ingestion was not halted. ‌ **Why it happened:** In a previous incident on February 6, 2023 \(more details at [https://status.mezmo.com/incidents/3yl9x1t7qcw5\),](https://status.mezmo.com/incidents/3yl9x1t7qcw5),) two pods storing logs were temporarily removed from the pool of pods available for receiving and inserting new batches of logs into our data store.  The pods continued to return results for previously processed logs.  We took this step because the pods had fallen behind on their tasks, which we believe was a consequence of an ungraceful shutdown during the incident.  We gave the pods several days to catch up on tasks and then made them available for insertion of new logs into the data store again, a change we expected to have no impact. One of the pods immediately began integrity checks to confirm the same data existed on its local disk and on our S3 storage.  As a side effect of the previous incident, the pod incorrectly determined that data was missing from the local disk and began sending http requests to our S3 storage to locate the missing data.  In fact, the data in question is designed to only reside on local disk and was not supposed to be stored on S3.     The requests failed with 404 errors when the data was not found on S3 \(as expected\).  Every new attempt to retrieve search results generated another request.  The rate of requests was high enough to slow down all requests related to search results within the pod’s zone \(one out of three total\).   This led to search results being returned slowly or not at all. ‌ **How we fixed it:** We removed the pod from the pool available for receiving and inserting new batches of logs into our data store.  The pod continued to return results for previously processed logs. ‌ **What we are doing to prevent it from happening again:** We marked this pod to remain unavailable for new logs until all previously processed logs on the pod have passed their retention period, whose maximum is 30 days.  At that time, the pod will be rebuilt and begin accepting newly submitted logs again. We’ll fix the logic of our search engine so it doesn’t request data from S3 that is intentionally not stored there. This will prevent the widespread 404 errors that slowed down all searching, should a pod again incorrectly determine it is missing data from its local disk. We have added alerting and monitoring to detect high latency in search speeds and the average time to compact newly inserted logs.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2023-02-10 18:11 UTC | Major incident: Searches are running slowly (this change) | degradation |

All Mezmo changes: [timeline](https://rewindex.com/company/mezmo/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i76e756fb46ca98 · Markdown: https://rewindex.com/change/i76e756fb46ca98.md · JSON: https://rewindex.com/api/v1/changes/i76e756fb46ca98 · Generated 2026-10-18T02:27:22.690Z