Changes / Braze / Currents and CDI latency impacting EU-01 and EU-02
Braze: Major incident: Currents and CDI latency impacting EU-01 and EU-02
Summary
Braze reported major Currents and CDI latency in EU-01 and EU-02 due to an infrastructure problem, starting 2025-03-10 15:47 UTC and resolved at 2025-03-10 22:21 UTC (6 h 33 min). New events and CDI syncs returned to normal, but the Currents event backlog was still being processed; outbound messaging operated normally.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
Major incidentLabel: degradation (model (Jev), 99%)Changed Detected Source: stspg.io (opens in new tab)
What changed
Before: not present
After: Currents and CDI latency impacting EU-01 and EU-02
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2025-03-10 15:47 UTC
Before: not present
After: 2025-03-10 22:21 UTC
Before: not present
After: Cloud Data-Ingestion (CDI), Currents, EU 01 Cluster - Cloud Data-Ingestion (CDI), EU 01 Cluster - Currents, EU 01 Cluster - Outbound Messaging, EU 02 Cluster - Cloud Data-Ingestion (CDI), EU 02 Cluster - Currents, EU 02 Cluster - Outbound Messaging, Outbound Messaging
Before: not present
After: We are currently experiencing latency issues with Currents and CDI due to an infrastructure problem. Our engineering team is actively working to resolve this and restore normal functionality as quickly as possible.
Before: not present
After: We continue to observe elevated latency impacting Currents and CDI. Our engineering teams are actively working to resolve this and restore normal functionality as quickly as possible.
Before: not present
After: We continue to observe elevated latency impacting Currents and CDI. Our engineering teams are actively working to resolve this and restore normal functionality as quickly as possible.
Before: not present
After: Engineers continue to work to resolve the issue impacting Currents and CDI.
Latency is currently highly elevated for Currents. Additionally, some customers may experience latency with CDI integrations.
Before: not present
After: Braze Engineers are actively working to resolve the ongoing issue impacting Currents and CDI. We are now observing elevated messaging latency as a result of this issue.
Currently, latency remains highly elevated for Currents, and some customers may also experience latency with CDI integrations.
We appreciate your patience as we work to resolve these issues.
Before: not present
After: The Braze Engineers have deployed changes to decrease Messaging latency and most messages are sent within normal operational bounds. Our Engineers continue to work on restoring the Currents and CDI Services. At this time Currents and CDI Services latency remains very high.
Before: not present
After: Outbound Messaging is operating within normal bounds following the changes deployed by The Braze Engineers. At this time Currents and CDI Services latency remains very high and our Engineers are continuing to work to restore the Currents and CDI Services.
Before: not present
After: Outbound Messaging is continuing to operate within normal bounds.
The Braze CDI is now fully operational, and all new events are being processed within normal operational bounds. If you do not regularly sync the CDI integration, you will need to replay any syncs that failed to fully complete. This step is necessary to prevent old data from overwriting new data that was successfully synced during the incident. You can check the status of your jobs in the dashboard under the Sync log tab within the Cloud Data Ingestion page.
Our Braze Engineers are actively working to restore the Currents service. We will provide further updates as the restoration progresses.
Before: not present
After: Outbound messaging continues to operate within normal bounds.
The Braze CDI continues to be fully operational, and all new events are being processed within normal operational bounds.
The Currents Service has been restored, and new events are being processed within normal operational bounds. Braze Engineers continue to work on processing the backlog of events impacted by the incident.
Before: not present
After: Currents is functioning normally, with all new events processing within normal operational bounds. Our Braze Engineering team is actively working to clear the backlog of Currents events.
Before: not present
After: We are continuing to monitor for any further issues.
Before: not present
After: As of 11:20am ET, both the Currents and Braze CDI services have been fully restored to normal operational performance. Since that time, all CDI syncs and new Currents events have been processing within standard parameters. The Braze Engineering team has completed the immediate restoration work and is continuing to process the backlog of Currents events produced during the incident's impact window. Depending on the volume of events and the downstream destination for your integration, the backlog processing may take some time and will be different for each integration. As such, with systems operating normally for new incoming events, we are marking this issue as resolved at this time. Should you have questions, please reach out to Braze Support.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Currents and CDI latency impacting EU-01 and EU-02 (this change) Braze reported major Currents and CDI latency in EU-01 and EU-02 due to an infrastructure problem, starting 2025-03-10 15:47 UTC and resolved at 2025-03-10 22:21 UTC (6 h 33 min). New events and CDI syncs returned to normal, but the Currents event backlog was still being processed; outbound messaging operated normally. | Major incident |