Changes / Opsgenie / Delays in Android push notifications
Opsgenie: Major incident: Delays in Android push notifications
Summary
Opsgenie reported a major incident delaying Android push notifications in the EU and US, starting 2023-04-04 14:30 UTC and resolved at 2023-04-04 18:16 UTC (3 h 45 min). The vendor attributed the delays to a Firebase error and confirmed Android push notifications were operational when Firebase’s incident was resolved.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Delays in Android push notifications
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2023-04-04 14:30 UTC
Before: not present
After: 2023-04-04 18:16 UTC
Before: not present
After: EU - Mobile Notification Delivery, Mobile Notification Delivery, US - Mobile Notification Delivery
Before: not present
After: We are investigating an issue with our Android push notifications that is impacting some of our notifications for Android. We will provide more details within the next hour.
Before: not present
After: The issue has been identified as caused by an error on Firebase (https://status.firebase.google.com/incidents/9ZPv9faHLen8bzLVSaft). We continue to monitor the situation and send update within the next hour.
Before: not present
After: Android push notification delivery is fully operational for now, but we are still monitoring the Firebase outage (https://status.firebase.google.com/incidents/9ZPv9faHLen8bzLVSaft).
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: This incident on Firebase has been resolved. Android push notifications are operational.
Before: not present
After: ### **SUMMARY**
On April 4, 2023, between 13:32 and 14:50 UTC, Atlassian customers using Opsgenie faced significant delays while receiving Android push notifications.
This was caused by an incident in a third party messaging service, which is responsible for Android push notification delivery. This in turn affected our systems.
The incident was immediately detected by our monitoring tools, our on-call engineers were paged, and at 14:50 UTC our systems recovered successfully.
The total time to resolution was about 80 minutes.
### **IMPACT**
The overall impact was between April 4, 2023, 13:32 - 14:50 UTC in Opsgenie_._ The incident only resulted in delays in Android push notifications only, and these notifications were delivered successfully after FCM service was restored and no data loss occurred.
### **ROOT CAUSE**
The issue was caused by an incident in a third party messaging service, which is responsible for delivering push notifications to Android devices.
**REMEDIAL ACTIONS PLAN & NEXT STEPS**
We know that outages impact your productivity. The impact was immediately caught by our monitoring tools, and the responsible team immediately started analysis of incident. We value transparency with our customers and will continue to notify you and take any necessary actions promptly during an incident.
In order to handle degradation or outage of messaging channels, Opsgenie recommends that users configure multiple channels of message delivery - including push notifications, mobile SMS, phone calls, and email.
In order to improve our response for the future, we will also be analyzing whether we can employ autoscaling solutions for our systems in case of an outage/high load related to one notification channel.
We apologize to customers whose services were impacted during this incident.
Thanks,
Atlassian Customer Support
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delays in Android push notifications (this change) Opsgenie reported a major incident delaying Android push notifications in the EU and US, starting 2023-04-04 14:30 UTC and resolved at 2023-04-04 18:16 UTC (3 h 45 min). The vendor attributed the delays to a Firebase error and confirmed Android push notifications were operational when Firebase’s incident was resolved. | Major incident |