Changes / Front / Delays in sending and receiving Gmail messages [gmail]
Front: Major incident: Delays in sending and receiving Gmail messages [gmail]
Summary
Front reported a major incident delaying Gmail message sending and receiving, initially in us-west-2 and later across multiple regions. It started at 2024-09-13 15:38 UTC and resolved at 2024-09-13 19:25 UTC, lasting 3 h 47 min. Front suspected Gmail rate limiting, drained the message backlog, and reported all channels back to normal.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 77%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Delays in sending and receiving Gmail messages [gmail]
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-09-13 15:38 UTC
Before: not present
After: 2024-09-13 19:25 UTC
Before: not present
After: Gmail
Before: not present
After: We're observing problems sending and receiving some messages on Gmail channels in the us-west-2 region. We're working on identifying the root cause.
Before: not present
After: We are continuing to investigate this issue, which seems to be related to Gmail's rate limiting mechanisms. We're seeing some impact across multiple regions.
Before: not present
After: We are continuing to see message delays, and are rescaling our internal queues globally to help mitigate the issue.
Before: not present
After: We are beginning to see a gradual increase in successful deliveries. We are continuing to monitor the situation, as well as replaying deliveries that have been delayed by the original incident.
Before: not present
After: We are on the recovery path now and are focused on draining the backlog of delayed messages.
Before: not present
After: We are continuing to drain the backlog and will be doing so for the next couple hours.
Before: not present
After: Continuing to drain the backlog
Before: not present
After: Some channels are still experiencing delays as we import the backlog of messages received while Gmail was experiencing an outage
Before: not present
After: All channels should be back to normal now.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Delays in sending and receiving Gmail messages [gmail] (this change) Front reported a major incident delaying Gmail message sending and receiving, initially in us-west-2 and later across multiple regions. It started at 2024-09-13 15:38 UTC and resolved at 2024-09-13 19:25 UTC, lasting 3 h 47 min. Front suspected Gmail rate limiting, drained the message backlog, and reported all channels back to normal. | Major incident |