# Mezmo: Major incident: Logs are not searchable in web app

**Summary:** Mezmo reported a major outage affecting Search, Livetail and the Web App: users could not search logs or use Live Tail in the Web UI. The incident ran from 2021-03-04 08:00 UTC to 2021-03-04 08:21 UTC (21 min); a fix was implemented at 2021-03-04 08:12 UTC. _(machine-written, sol)_
- Kind: Major incident · Label: outage (model (Jev))
- Changed: 2021-03-04 08:00 UTC · Detected: 2026-10-17 18:22 UTC
- [Source: https://stspg.io/r3k0jq9vr5cr](https://stspg.io/r3k0jq9vr5cr)
- [Company: Mezmo](https://rewindex.com/company/mezmo.md)
- Versions: before (none: first archived) → after incident\_version:282/8hl52136k3g0@79e43a1bb33f
- Change 1 of 1 for this incident

## What changed

**1. Added: Title**

```diff
+ Logs are not searchable in web app
```

**2. Added: Impact**

```diff
+ major
```

**3. Added: Status**

```diff
+ postmortem
```

**4. Added: Start time**

```diff
+ 2021-03-04 08:00 UTC
```

**5. Added: Resolved time**

```diff
+ 2021-03-04 08:21 UTC
```

**6. Added: Affected components**

```diff
+ Livetail, Search, Web App
```

**7. Added: Update** (investigating) · 2021-03-04 08:00 UTC

```diff
+ We are currently investigating an issue that is rendering our log viewer unavailable at this time.
```

**8. Added: Update** (monitoring) · 2021-03-04 08:12 UTC

```diff
+ A fix has been implemented and we are monitoring the results.
```

**9. Added: Update** (resolved) · 2021-03-04 08:21 UTC

```diff
+ This incident has been resolved and logs are searchable in the web app.  We'll continue to monitor all services.
```

**10. Added: Update** (postmortem) · 2021-03-25 17:30 UTC

```diff
+ **Dates:**   Start Time: Thursday, March 4, 2021, at ~03:45 UTC   End Time:  Thursday, March 4, 2021, at ~08:20 UTC   Duration: ~4:36:00 ‌ **What happened:** Our Web UI returned an error message "Request returned an error. Try again?" when users tried to perform a search query or use Live Tail in the Web UI. ‌ **Why it happened:** The pods that run our searching and Live Tail services were automatically terminated by our Kubernetes orchestration system.  Upon investigation, we discovered we had inadvertently classed these services as low priority.  The incident occurred when a large number of other services that were classed as higher priority needed to run to meet usage demands.  The orchestration system automatically terminated the lower priority services to make resources available for the higher priority services. More specifically, these pods were put into a “terminating” state.  Normally this state is temporary -- a transition between “running” and “terminated”.  During this incident, the pods remained in the “terminating” state permanently.  Our monitoring detects services that have been “terminated”, but not ones that are in the temporary “terminating” state.  Consequently, our infrastructure team was not notified. ‌ **How we fixed it:** We increased the priority of the pods that run our searching and Live Tail services to match the priority of other services.  We updated the configuration of our orchestration system to make the change permanent. ‌ **What we are doing to prevent it from happening again:** We’ve already updated the configuration of our orchestration system to give services the correct priority.  These changes are permanent and should prevent similar problems in the future.
```

## History of this incident

| When (UTC) | Change | Label |
| --- | --- | --- |
| 2021-03-04 08:00 UTC | Major incident: Logs are not searchable in web app (this change) | outage |

All Mezmo changes: [timeline](https://rewindex.com/company/mezmo/changes.md)
_Diff excerpts are shown on normalized text: dates, counters and tokens appear as ‹date›, ‹count›, ‹tok›; noise-only edits are hidden. Rewindex shows the changed passages, not a copy of the page._


---
Canonical: https://rewindex.com/change/i602a5e1f1b1ee8 · Markdown: https://rewindex.com/change/i602a5e1f1b1ee8.md · JSON: https://rewindex.com/api/v1/changes/i602a5e1f1b1ee8 · Generated 2026-10-18T03:33:58.297Z