# Mezmo: Search and Live Tail Intermittent Failures

- Impact (vendor’s label): major
- Status: postmortem
- Started: 2021-08-19 13:56 UTC · Resolved: 2021-08-19 20:48 UTC · Duration: 6.9 h
- Updates: 5 · Affected components: Search, Livetail
- [Vendor status page: https://stspg.io/frbwvqpqrc2f](https://stspg.io/frbwvqpqrc2f)
- [Company: Mezmo](https://rewindex.com/company/mezmo.md)

## Summary

Mezmo reported a major incident affecting Web UI searches and Live Tail from 2021-08-19 13:56 UTC to 2021-08-19 20:48 UTC, lasting 6 h 52 min. Intermittent failures affected some customers; about 12% also experienced delays before newly submitted logs were available for searching, graphing, and timelines. Mezmo took remedial action and reported all services fully operational. _(machine-written, sol)_
## Timeline of updates

- **investigating** 2021-08-19 13:56 UTC: Searches in our Web UI and Live Tail are failing intermittently, for some customers. We are investigating.
- **investigating** 2021-08-19 15:31 UTC: Our engineers are taking steps to mitigate the impact as we identify the root cause.
- **identified** 2021-08-19 17:30 UTC: We’ve identified the immediate cause and are taking remedial action.  Searches in our Web UI and Live Tail are working better now, but are still failing at times.
- **resolved** 2021-08-19 20:48 UTC: Searching and live tail are working as expected. All services are fully operational.
- **postmortem** 2021-08-20 17:06 UTC: \*\*Start Time:\*\* Thursday, August 19, 2021, at, 13:56 UTC \*\*End Time:\*\* Thursday, August 19, 2021, at, 20:48 UTC \*\*Duration:\*\* 6:52:00      \*\*What happened:\*\* Searches in our Web UI and Live Tail were intermittently failing.  Additionally, for a small set of customers \\(about 12%\\), there were delays in newly submitted logs being available for searching, graphing, and timelines.       \*\*Why it happened:\*\* Our service uses the Calico networking solution to ensure network level connectivity between all nodes and the pods running on them.  On several nodes, Calico stopped running.  This put one of our ElasticSearch clusters into an unhealthy red state.  For customers using this cluster \\(about 12% of all our customers\\), there were delays in newly submitted logs being available for searching, graphing, and timelines.  When Calico stopped running on some nodes, it also led to failures with tribe nodes, which make searching across multiple clusters possible.  This caused intermittent failures in searching and live tail, for all customers. ‌ \*\*How we fixed it:\*\* We took remedial action by restarting Calico and restoring networking connections between all nodes.  We also restarted our tribe nodes, repaired the ElasticSearch cluster that was in a red state, and then provided temporary resources so our service could more quickly process the backlog of logs sent by customers since the beginning of the incident. ‌ \*\*What we are doing to prevent it from happening again:\*\* We are investigating why Calico stopped working on several nodes.  We’re also updating our runbooks to recover more quickly in similar situations and limit any customer impact.

## Rewindex changes for this incident

- 2021-08-19 13:56 UTC: [Major incident: Search and Live Tail Intermittent Failures](https://rewindex.com/change/ice594024d05d2d.md)


---
Canonical: https://rewindex.com/company/mezmo/incidents/ytc4r5532n90 · Markdown: https://rewindex.com/company/mezmo/incidents/ytc4r5532n90.md · JSON: https://rewindex.com/api/v1/companies/mezmo/incidents/ytc4r5532n90 · Generated 2026-10-18T12:41:38.604Z