Search

Companies / Mezmo / incidents

Mezmo: Ingestion, Searching, Live Tail, Alerting, Graphing, and Timelines Delays

Impact: criticalStatus: postmortemStarted Resolved (1.1 h)5 updatesSource: stspg.io (opens in new tab)
Summary

Mezmo reported a critical outage: ingestion halted, delaying Searching, Live Tail, Alerting, Graphing, and Timelines; some alerts never triggered. The incident ran from 2022-01-26 16:10 UTC to 2022-01-26 17:15 UTC, lasting 1 h 5 min. A fix restored ingestion and all services by 16:58 UTC.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Web AppLog Ingestion (Agent/REST API/Code Libraries)Log Ingestion (Syslog)SearchAlertingLog Ingestion (Heroku)Livetail

Timeline of updates

  1. investigating

    Ingestion services are currently halted. Customers will also experience delays with Searching, Live Tail, Alerting, Graphing, and Timelines.

  2. investigating

    We are continuing to investigate this issue.

  3. monitoring

    We have implemented a fix and are monitoring the results. Logs are being ingested again and all services are operational.

  4. resolved

    This incident has been resolved. All services are operational.

  5. postmortem

    **Dates:** Start Time: Wednesday, January 26, 2022, at 15:45:00 UTC End Time: Wednesday, January 26, 2022, at 16:30:00 UTC Duration: 00:45:00 ‌ **What happened:** Ingestion was halted and newly submitted logs were not immediately available for Alerting, Live Tail, Searching, Graphing, and Timelines. Some alerts were never triggered. Once ingestion had resumed, LogDNA agents running on customer environments resent all locally cached logs to our service for ingestion. No data was lost. ‌ **Why it happened:** Our Redis database had a failover and the services that depend on it were unable to recover automatically. Normally, the pods running our ingestion service deliberately crash until they are able to access Redis again. However, these pods were in a bad state and unable to reconnect when Redis returned. Since ingestion was halted, newly submitted logs were not passed on to many downstream services, such as Alerting, Live Tail, Searching, Graphing, and Timelines. ‌ **How we fixed it:** We manually restarted all the pods of our ingestion service, then restarted all the sentinel pods of Redis. The ingestion service became operational again and logs were passed on to all downstream services. Over a short period of time, these services processed the backlog of logs and newly submitted logs were again available without delays. ‌ **What we are doing to prevent it from happening again:** The ingestion pods were in a bad state because they had not been restarted after a configuration change made several days earlier, for reasons unrelated to this incident. The runbook for making such configuration changes has been updated to prevent this procedural failure in the future. We’re also in the middle of a project to make all services more tolerant of Redis failovers.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Mezmo profile · Recent Mezmo incidents · Markdown · JSON