Changes / Datadog / Delayed APM metric ingestion
Datadog: Major incident: Delayed APM metric ingestion
Summary
Datadog reported a major APM and process metric ingestion delay starting 2025-12-12 18:37 UTC, affecting monitor evaluation and causing some incorrect alerts. The incident was resolved at 2025-12-12 21:52 UTC after 3 h 15 min; APM impact ended, while remaining live process data impact was tracked in a separate incident.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Delayed APM metric ingestion
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-12-12 18:37 UTC
Before: not present
After: 2025-12-12 21:52 UTC
Before: not present
After: Metrics and Infra Monitoring
Before: not present
After: We are currently investigating lag in ingesting apm metrics, which affects monitor evaluation.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We are currently investigating lag in ingesting apm and process metrics, which affects monitor evaluation and in some cases led to incorrect monitor alerts.
Before: not present
After: We have identified the issue affecting ingestion delays in apm and process metrics and are working on recovery
Before: not present
After: All impact related to APM metrics has been resolved. A separate incident has been created to track the remaining impact in live process data.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delayed APM metric ingestion (this change) Datadog reported a major APM and process metric ingestion delay starting 2025-12-12 18:37 UTC, affecting monitor evaluation and causing some incorrect alerts. The incident was resolved at 2025-12-12 21:52 UTC after 3 h 15 min; APM impact ended, while remaining live process data impact was tracked in a separate incident. | Major incident |