Changes / Rippling / Notification delay
Rippling: Major incident: Notification delay
Summary
Rippling reported a major notification degradation causing 2–15 minute authentication delays, starting 2025-10-27 15:23 UTC and resolved at 2025-10-27 18:38 UTC (3 h 14 min). Scheduled notifications were temporarily suspended to reduce database load; delivery recovered within the 10-second SLA, scheduled jobs resumed, and delayed scheduled notifications were being sent.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 99%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Notification delay
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-10-27 15:23 UTC
Before: not present
After: 2025-10-27 18:38 UTC
Before: not present
After: Notifications, Platform Tools - Notifications, Rippling App
Before: not present
After: Customers may be experiencing a delay in notifications causing authentication delays. We are observing a 2 - 15 minute delay in notifications. This is under investigation.
Before: not present
After: We are temporarily suspending scheduled notification jobs in order to bring down the database load so that all other notifications -- including time-sensitive authentication notifications -- can recover to be delivered within SLA.
Before: not present
After: We have partial recovery where new notifications are being sent within the SLA of 10 seconds. Scheduled notifications continue to be suspended.
Before: not present
After: The notification service has been restored, and notifications are now properly getting sent within the SLA. Scheduled notification jobs have been restored, so moving forward they will be sent at the scheduled time.
Scheduled notifications that were scheduled to be sent during the suspended period -- between 9:15a PDT to 11:23a PDT -- are being sent now.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Notification delay (this change) Rippling reported a major notification degradation causing 2–15 minute authentication delays, starting 2025-10-27 15:23 UTC and resolved at 2025-10-27 18:38 UTC (3 h 14 min). Scheduled notifications were temporarily suspended to reduce database load; delivery recovered within the 10-second SLA, scheduled jobs resumed, and delayed scheduled notifications were being sent. | Major incident |