Changes / Pusher / We are experiencing a major outage in our US2 cluster

Pusher: Critical incident: We are experiencing a major outage in our US2 cluster

Summary

Summary pending. The diff below shows exactly what changed.

Major incidentLabel: outage (model (Jev), 100%)Changed Detected Source: stspg.io

What changed

AddedTitle#1
Before: not present
After: We are experiencing a major outage in our US2 cluster
AddedImpact#2
Before: not present
After: critical
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2024-09-08 18:00 UTC
AddedResolved time#5
Before: not present
After: 2024-09-08 23:06 UTC
AddedAffected components#6
Before: not present
After: Channels presence channels, Channels REST API, Channels Webhooks, Channels WebSocket client API
AddedUpdateinvestigating#7
Before: not present
After: We are investigating what is causing the issue to our US cluster.
AddedUpdateinvestigating#8
Before: not present
After: We are escalating the current issue from a partial outage to a major outage. The majority of traffic in the US2 cluster is currently affected. Our team is fully engaged and working to resolve the issue.
AddedUpdateinvestigating#9
Before: not present
After: Our team is still investigating the issue within our infrastructure that is preventing us from adding WebSocket capacity to the US2 cluster. This issue is specifically impacting our WebSocket deployments. The majority of clients are currently unable to connect using WebSocket. However fallback mechanisms (SockJS, HTTP streaming, and HTTP polling) are available for connection if they are not disabled in your configuration.
AddedUpdatemonitoring#10
Before: not present
After: The team has been able to identify the issue and push a fix. We are seeing a full recovery of the services. We will continue to monitor.
AddedUpdatemonitoring#11
Before: not present
After: A fix has been implemented, and we are monitoring the results. WebSocket connections should now be functional.
AddedUpdatemonitoring#12
Before: not present
After: We are continuing to monitor for any further issues.
AddedUpdateresolved#13
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident.

History of this incident

When (UTC)ChangeKind
Critical incident: We are experiencing a major outage in our US2 cluster (this change)Major incident

All Pusher changes → · Pusher profile · Markdown · JSON