Front: Delays in sending and receiving Gmail messages [gmail]
Front reported a major incident delaying Gmail message sending and receiving, initially in us-west-2 and later across multiple regions. It started at 2024-09-13 15:38 UTC and resolved at 2024-09-13 19:25 UTC, lasting 3 h 47 min. Front suspected Gmail rate limiting, drained the message backlog, and reported all channels back to normal.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Gmail
Timeline of updates
- investigating
We're observing problems sending and receiving some messages on Gmail channels in the us-west-2 region. We're working on identifying the root cause.
- investigating
We are continuing to investigate this issue, which seems to be related to Gmail's rate limiting mechanisms. We're seeing some impact across multiple regions.
- investigating
We are continuing to see message delays, and are rescaling our internal queues globally to help mitigate the issue.
- investigating
We are beginning to see a gradual increase in successful deliveries. We are continuing to monitor the situation, as well as replaying deliveries that have been delayed by the original incident.
- identified
We are on the recovery path now and are focused on draining the backlog of delayed messages.
- monitoring
We are continuing to drain the backlog and will be doing so for the next couple hours.
- monitoring
Continuing to drain the backlog
- monitoring
Some channels are still experiencing delays as we import the backlog of messages received while Gmail was experiencing an outage
- resolved
All channels should be back to normal now.
Rewindex changes for this incident
- Major incident: Delays in sending and receiving Gmail messages [gmail] 2024-09-13 15:38 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Front profile · Recent Front incidents · Markdown · JSON