Changes / Pusher / US3 cluster - major outage
Pusher: Major incident: US3 cluster - major outage
Summary
Pusher reported a major US3 outage affecting Channels presence channels, Channels REST API and Channels WebSocket client API, with latency, 503 errors and full service downtime. It started at 2025-10-20 18:36 UTC and resolved at 2025-10-20 20:41 UTC, lasting 2 h 5 min. The postmortem identifies Redis cluster startup failures; Pusher reported implementing a fix.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: US3 cluster - major outage
Before: not present
After: major
Before: not present
After: postmortem
Before: not present
After: 2025-10-20 18:36 UTC
Before: not present
After: 2025-10-20 20:41 UTC
Before: not present
After: Channels presence channels, Channels REST API, Channels WebSocket client API
Before: not present
After: We're seeing an increased latency and a number of 503 errors on us3 cluster. The team is investigating.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We are continuing to investigate this issue.
Before: not present
After: We are currently experiencing a major outage in US3 cluster. Our engineering team is actively working to restore full functionality as quickly as possible. In the meantime, we recommend customers temporarily switch to an alternative cluster to minimize service disruption.
Before: not present
After: Our team has identified the root cause of the issue and implemented a fix. The affected cluster is now in the process of recovering, and performance is gradually returning to normal. We will continue to closely monitor the situation until full recovery is confirmed.
Before: not present
After: All services are now fully operational. Our team continues to monitor the system closely to ensure ongoing stability and performance.
Before: not present
After: This incident has been resolved.
Before: not present
After: ## Root Cause Analysis: Redis Cluster Startup Failures – US3 Cluster
**Incident Date:** October 20, 2025
**Duration:** 18:36 UTC – 20:41 UTC
**Status:** Resolved
## **Summary**
On October 20, 2025, customers experienced a significant outage affecting the US3 cluster.
The disruption began with increased latency and 503 errors before escalating into full service downtime as Redis clusters in the US3 region failed to start successfully during an infrastructure update.
Service was fully restored at 20:41 UTC after engineers identified and resolved the underlying startup issue, confirming stability across all Redis clusters.
## **Impact**
Between **18:36 UTC and 20:41 UTC**, customers experienced:
* Major outage in the **US3** cluster, impacting message delivery and connection reliability
* Elevated error rates and timeouts across dependent APIs
* Temporary need for customers to switch to alternate clusters to maintain service continuity
No customer data was lost. However, applications relying solely on the affected cluster experienced full downtime for a large portion of the incident.
## **Root Cause**
The outage was caused by a failure of Redis clusters to start correctly following an infrastructure update due to a missing configuration flag upon startup.
An unexpected upgrade to the Docker runtime running our Redis cluster introduced a breaking change that prevented container startup for certain Redis deployments. When replacement Redis instances in the US3 redis-main cluster were launched, they failed initialization checks and repeatedly restarted, rendering the cluster unavailable.
The incompatibility remained undetected until a routine node replacement in the US3 cluster introduced new Redis instances to the cluster.
## **Detection and Response**
Monitoring systems first detected increased error rates and latency at **18:36 UTC**, followed by a rise in 503 responses from the affected APIs.
**Timeline of events:**
* **18:36 UTC** – Increased latency and 503 errors observed in US3 cluster
* **19:01 UTC** – Engineering began investigation into Redis startup failures
* **19:04 UTC** – Incident declared a major outage; mitigation efforts initiated
* **19:59 UTC** – Root cause identified and configuration fix applied
* **20:20 UTC** – Services operational; monitoring for recovery stability
* **20:41 UTC** – All Redis nodes confirmed healthy; incident resolved
## **Resolution**
The engineering team:
* Implemented a temporary fix to the Redis environments ensure Redis instances could initialize successfully
* Blocked further automated replacements in other clusters until validated
* Verified recovery and stability across all Redis clusters
After deployment of the fix, Redis instances in the US3 redis-main cluster started correctly and full service was restored.
## **Preventative Actions**
To prevent recurrence, the team has:
* Rolled out a permanent fix across all Redis clusters used in all regions
* Planned a long-term remediation to modernize Redis image packaging for compatibility with current and future Docker releases
## **Next Steps and Commitment**
We are conducting a broader review of infrastructure upgrade processes to better detect runtime incompatibilities before they impact production.
We apologize for the disruption this incident caused. Ensuring reliability and transparency remains our highest priority, and we continue to strengthen our processes to maintain consistent, predictable service for all Pusher customers.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: US3 cluster - major outage (this change) Pusher reported a major US3 outage affecting Channels presence channels, Channels REST API and Channels WebSocket client API, with latency, 503 errors and full service downtime. It started at 2025-10-20 18:36 UTC and resolved at 2025-10-20 20:41 UTC, lasting 2 h 5 min. The postmortem identifies Redis cluster startup failures; Pusher reported implementing a fix. | Major incident |