Datadog: Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered

Impact: majorStatus: resolvedStarted 2026-04-10 14:54 UTCResolved 2026-04-10 20:59 UTC (6.1 h)14 updatesSource: stspg.io

Summary

Datadog reported a major Monitors notification degradation in US1, AP1, and AP2 affecting PagerDuty, Slack, Microsoft Teams, Webhooks, Jira, ServiceNow, OpsGenie, VictorOps, BigPanda, Zendesk, and Sumologic. The incident ran from 2026-04-10 14:54 UTC to 2026-04-10 20:59 UTC (6 h 5 min); notifications were delayed or temporarily undelivered, and Datadog reported replaying all delayed notifications.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Monitors

Timeline of updates

investigating2026-04-10 14:54 UTC
We are investigating an issue to send notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie.
identified2026-04-10 15:00 UTC
The issue has been identified and a fix is being implemented.
identified2026-04-10 15:23 UTC
We are continuing to work on a fix for this issue.
identified2026-04-10 15:42 UTC
New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed.
monitoring2026-04-10 15:42 UTC
New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed
monitoring2026-04-10 15:59 UTC
We are still catching-up on past notifications that may have been delayed.
monitoring2026-04-10 16:15 UTC
We are still catching-up on past notifications that may have been delayed.
monitoring2026-04-10 17:12 UTC
Notifications for Opsgenie, VictorOps, BigPanda, Zendesk, and Sumologic are caught up. Webhooks, Microsoft Teams, Slack, JIRA, and ServiceNow are still catching up.
monitoring2026-04-10 17:37 UTC
Notifications for JIRA are now caught up.
monitoring2026-04-10 18:33 UTC
Notifications for ServiceNow and Microsoft Teams are now caught up. Webhooks and Slack notifications are still catching up.
monitoring2026-04-10 19:02 UTC
Webhooks and Slack notifications are still catching up.
monitoring2026-04-10 19:44 UTC
Webhooks and Slack notifications are still catching up.
monitoring2026-04-10 20:13 UTC
Webhooks and Slack notifications are still catching up.
resolved2026-04-10 20:59 UTC
Between approximately 9:42 AM and 10:38 AM ET on April 10, we observed delivery failures for customers in the US1, AP1, and AP2 regions. During this window, notifications to the following integrations were delayed or temporarily undelivered: • PagerDuty • Slack • Microsoft Teams • Webhooks • Jira • ServiceNow • OpsGenie • VictorOps • BigPanda • Zendesk • Sumologic All delayed notifications have been replayed with the following exceptions: PagerDuty pages and Slack notifications queued during the outage window were not replayed, as they were no longer actionable at the time of recovery. Webhook notifications will be reprocessed and delivered at a later date. No monitoring data was lost during this incident. Monitor evaluations continued normally — only the delivery of triggered notifications was affected. Thank you for your patience.

Rewindex changes for this incident

Major incident: Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered 2026-04-10 14:54 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Datadog profile · Recent Datadog incidents · Markdown · JSON