Changes / Mezmo / Web UI unavailable and ingestion has stopped
Mezmo: Critical incident: Web UI unavailable and ingestion has stopped
Summary
Mezmo reported a critical outage affecting the Web UI and log ingestion, starting at 2021-01-14 20:05 UTC and resolved at 2021-01-14 20:27 UTC (computed duration: 21 min). All services were restored; the postmortem separately reports 45 minutes of disruption and says ingestion client agent logs were automatically resent and successfully ingested.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Web UI unavailable and ingestion has stopped
Before: not present
After: critical
Before: not present
After: postmortem
Before: not present
After: 2021-01-14 20:05 UTC
Before: not present
After: 2021-01-14 20:27 UTC
Before: not present
After: Log Ingestion (Agent/REST API/Code Libraries), Log Ingestion (Heroku), Log Ingestion (Syslog), Web App
Before: not present
After: The web UI is unavailable and ingestion has stopped. We are investigating.
Before: not present
After: The web UI is available again and ingestion has resumed. All services are operational.
Before: not present
After: **Dates:**
Start Time: Thursday, January 14, 2021, at 19:42 UTC
End Time: Thursday, January 14, 2021, at 20:27 UTC
Duration: 0:45:00
**What happened:**
Our WebUI became unavailable and ingestion of new logs stopped for 45 minutes. Logs were automatically resent later and ingested successfully for customers using our ingestion client agent.
**Why it happened:**
The certificate used by all our services expired. Consequently, all API calls to our service failed, which caused our WebUI to fail and ingestion of new logs to stop.
**How we fixed it:**
We renewed the certificate and applied it to all affected services. Our WebUI became responsive again and ingestion resumed. Since no logs had been ingested for about 45 minutes, our service had a moderately large backlog to process. As it caught up, users experienced delays in searching, graphing, and timelines for newly submitted logs.
**What we are doing to prevent it from happening again:**
We’re tightening our internal notifications of upcoming expiration dates for all certificates our service relies upon.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Web UI unavailable and ingestion has stopped (this change) Mezmo reported a critical outage affecting the Web UI and log ingestion, starting at 2021-01-14 20:05 UTC and resolved at 2021-01-14 20:27 UTC (computed duration: 21 min). All services were restored; the postmortem separately reports 45 minutes of disruption and says ingestion client agent logs were automatically resent and successfully ingested. | Major incident |