Search

Companies / Braze / incidents

Braze: Widespread Internet Issue

Impact: majorStatus: resolvedStarted Resolved (2.5 h)10 updatesSource: stspg.io (opens in new tab)
Summary

Braze reported major degradation from 2025-06-12 19:01 UTC to resolution at 2025-06-12 21:30 UTC: message sending and data processing latency, SDK API errors, and disrupted Android push delivery. The Google Cloud outage prevented Kubernetes scaling; infrastructure later scaled up to process backlogs, and Google confirmed Android Push / Firebase messaging recovery.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: SDK Data CollectionUS 06 Cluster - SDK Data CollectionEU 01 Cluster - SDK Data CollectionSDK Data CollectionDashboardUS 06 Cluster - DashboardEU 01 Cluster - REST APIsREST APIsREST APIsUS 07 Cluster - REST APIsData ProcessingUS 02 Cluster - Data ProcessingSDK Data CollectionUS 02 Cluster - SDK Data CollectionDashboardUS 10 Cluster - DashboardData ProcessingUS 03 Cluster - Data ProcessingEU 02 Cluster - REST APIsREST APIsOutbound MessagingUS 07 Cluster - Outbound MessagingData ProcessingEU 01 Cluster - Data ProcessingREST APIsUS 02 Cluster - REST APIsData ProcessingEU 02 Cluster - Data ProcessingData ProcessingUS 01 Cluster - Data ProcessingSDK Data CollectionUS 10 Cluster - SDK Data CollectionREST APIsUS 05 Cluster - Rest APIsDashboardUS 02 Cluster - DashboardREST APIsUS 04 Cluster - REST APIsDashboardEU 01 Cluster - DashboardDashboardUS 04 Cluster - DashboardREST APIsUS 06 Cluster - REST APIsOutbound MessagingUS 10 Cluster - Outbound MessagingOutbound MessagingUS 04 Cluster - Outbound MessagingDashboardUS 05 Cluster - DashboardData ProcessingUS 10 Cluster - Data ProcessingEU 02 Cluster - Outbound MessagingOutbound MessagingREST APIsUS 03 Cluster - REST APIsREST APIsUS 01 Cluster - REST APIsData ProcessingUS 04 Cluster - Data ProcessingSDK Data CollectionUS 01 Cluster - SDK Data CollectionEU 01 Cluster - Outbound MessagingOutbound MessagingSDK Data CollectionUS 05 Cluster - SDK Data CollectionOutbound MessagingUS 06 Cluster - Outbound MessagingOutbound MessagingUS 01 Cluster - Outbound MessagingDashboardEU 02 Cluster - DashboardSDK Data CollectionUS 04 Cluster - SDK Data CollectionDashboardUS 07 Cluster - DashboardOutbound MessagingUS 03 Cluster - Outbound MessagingData ProcessingUS 06 Cluster - Data ProcessingSDK Data CollectionUS 07 Cluster - SDK Data CollectionGlobal Messaging Channels - Push Notifications - AndroidPush Notifications - AndroidDashboardUS 01 Cluster - DashboardData ProcessingUS 07 Cluster - Data ProcessingSDK Data CollectionUS 03 Cluster - SDK Data CollectionOutbound MessagingUS 02 Cluster - Outbound MessagingREST APIsUS 10 Cluster - REST APIsDashboardUS 03 Cluster - DashboardData ProcessingUS 05 Cluster - Data ProcessingEU 02 Cluster - SDK Data CollectionSDK Data CollectionOutbound MessagingUS 05 Cluster - Outbound Messaging

Timeline of updates

  1. investigating

    There appears to be a widespread internet issue affecting multiple services around the internet. Cloudflare, a Braze sub-processor, is having an incident (https://www.cloudflarestatus.com/incidents/25r9t0vz99rp) and Google Cloud is having a major outage (https://status.cloud.google.com/). This is affecting our ability to deliver push notifications via Google Firebase. AWS is not provisioning new servers, which is preventing our ability to scale up, and may be related to this overall internet issue. We will provide additional updates soon.

  2. identified

    The Google Cloud outage is preventing our ability to scale up our kubernetes clusters. As a result, customers will experience message sending and data processing latency until this recovers. We are also seeing SDK API errors across multiple clusters.

  3. identified

    Multiple Braze Services are experiencing degradation due to the aforementioned incident with Cloudflare and Google Cloud Platform. While all services are available and operating correctly, we are unable to scale up our infrastructure and have taken some precautions to prevent a cascading failure once the Google and Cloudflare incidents resolve and traffic starts flowing at higher rates again. Our SDK API is mostly returning error response codes to SDKs–this directs them to retry requests at a later time to throttle processing demand. Events and attribute updates will not be lost, and our SDKs will periodically retry requests automatically. Our REST APIs are operational; however, as inbound internet traffic is challenged, some requests may have timed out before they get to Braze. Per standard practice, any REST API calls that timeout or receive a 50X response code should be retried. Dashboard users may experience sporadic login timeouts when visiting https://dashboard.braze.com until upstream providers recover. Outbound messaging and data processing are continuing, but may experience latency until the incident is fully resolved upstream and Braze can re-scale our infrastructure. We are starting to see some signs of recovery and will be proactively ramping services back to normal behavior as we are able to. We will also continue to post regular updates.

  4. identified

    Our SDK API has resumed normal processing. Events and attribute updates will not be lost; we now are automatically retrying and processing those events. Dashboard login services have now recovered, and users can log in through https://dashboard.braze.com.

  5. identified

    We are now actively scaling up our infrastructure and expect to return to normal performance soon. As this happens, the Outbound Messaging and Data Processing backlog is being worked on. We expect latency to dissipate as we process through the queue.

  6. identified

    We continue to see signs of recovery and are ramping services back to normal as we are able to. Our REST APIs are operational; however, as some inbound internet traffic remains challenged, some requests may have timed out before they get to Braze. Per standard practice, any REST API calls that timeout or receive a 50X response code should be retried.

  7. identified

    Our infrastructure has fully scaled up, so we are processing well above our typical speeds to work through the backlogs. Several of our clusters are now fully operational, as the Data Processing and Outbound Messaging backlogs have been fully processed. US-01, US-02, US-03, US-04, US-05, US-07, and EU-02 are fully recovered.

  8. identified

    Google has confirmed that Android Push / Firebase messaging is fully operational, and backlogs have been processed.

  9. identified

    US-06 is now fully recovered. Data Processing and Outbound Messaging backlogs have been fully processed.

  10. resolved

    EU-01 is now fully recovered. Data Processing and Outbound Messaging backlogs have been fully processed. With that, Braze is fully operational.

Rewindex changes for this incident

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Braze profile · Recent Braze incidents · Markdown · JSON