Changes / Datadog / Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered
Datadog: Major incident: Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered
Summary
Datadog reported a major Monitors notification degradation in US1, AP1, and AP2 affecting PagerDuty, Slack, Microsoft Teams, Webhooks, Jira, ServiceNow, OpsGenie, VictorOps, BigPanda, Zendesk, and Sumologic. The incident ran from 2026-04-10 14:54 UTC to 2026-04-10 20:59 UTC (6 h 5 min); notifications were delayed or temporarily undelivered, and Datadog reported replaying all delayed notifications.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 74%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2026-04-10 14:54 UTC
Before: not present
After: 2026-04-10 20:59 UTC
Before: not present
After: Monitors
Before: not present
After: We are investigating an issue to send notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie.
Before: not present
After: The issue has been identified and a fix is being implemented.
Before: not present
After: We are continuing to work on a fix for this issue.
Before: not present
After: New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed.
Before: not present
After: New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed
Before: not present
After: We are still catching-up on past notifications that may have been delayed.
Before: not present
After: We are still catching-up on past notifications that may have been delayed.
Before: not present
After: Notifications for Opsgenie, VictorOps, BigPanda, Zendesk, and Sumologic are caught up. Webhooks, Microsoft Teams, Slack, JIRA, and ServiceNow are still catching up.
Before: not present
After: Notifications for JIRA are now caught up.
Before: not present
After: Notifications for ServiceNow and Microsoft Teams are now caught up. Webhooks and Slack notifications are still catching up.
Before: not present
After: Webhooks and Slack notifications are still catching up.
Before: not present
After: Webhooks and Slack notifications are still catching up.
Before: not present
After: Webhooks and Slack notifications are still catching up.
Before: not present
After: Between approximately 9:42 AM and 10:38 AM ET on April 10, we observed delivery failures for customers in the US1, AP1, and AP2 regions. During this window, notifications to the following integrations were delayed or temporarily undelivered:
• PagerDuty
• Slack
• Microsoft Teams
• Webhooks
• Jira
• ServiceNow
• OpsGenie
• VictorOps
• BigPanda
• Zendesk
• Sumologic
All delayed notifications have been replayed with the following exceptions: PagerDuty pages and Slack notifications queued during the outage window were not replayed, as they were no longer actionable at the time of recovery. Webhook notifications will be reprocessed and delivered at a later date.
No monitoring data was lost during this incident. Monitor evaluations continued normally — only the delivery of triggered notifications was affected.
Thank you for your patience.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered (this change) Datadog reported a major Monitors notification degradation in US1, AP1, and AP2 affecting PagerDuty, Slack, Microsoft Teams, Webhooks, Jira, ServiceNow, OpsGenie, VictorOps, BigPanda, Zendesk, and Sumologic. The incident ran from 2026-04-10 14:54 UTC to 2026-04-10 20:59 UTC (6 h 5 min); notifications were delayed or temporarily undelivered, and Datadog reported replaying all delayed notifications. | Major incident |