Search

Companies / RudderStack / incidents

RudderStack: Investigating elevated delivery failures for some destinations

Impact: minorStatus: resolvedStarted Resolved (7.2 h)6 updatesSource: stspg.io (opens in new tab)

Affected components: Data Processing Pipeline - Data Processing Pipeline (US)Data Processing Pipeline (US)Data Processing Pipeline - Data Processing Pipeline (EU)Data Processing Pipeline (EU)

Timeline of updates

  1. investigating

    We are investigating an increase in delivery failures (authentication errors) affecting a subset of Webhook destinations for some customers, beginning around 06:35 UTC on July 2. Event ingestion is not affected. We are actively investigating and will provide an update shortly.

  2. identified

    Elevated delivery failures for Webhook destinations We've identified the root cause of the delivery failures (authentication errors / HTTP 401) affecting some Webhook destinations. Event ingestion is unaffected — your data is being received and processed normally; only delivery to these destinations is impacted. Our engineering team is actively working on a fix and remediation is underway. We will post the next update as soon as recovery is confirmed. Immediate fix on your side: Open your Webhook destination settings in the dashboard, re-enter the keys/values in the Headers section (the previous values were reset), and save. Event delivery will resume right away.

  3. identified

    We are continuing to work on a fix for this issue.

  4. identified

    We are continuing to work on a fix for this issue.

  5. identified

    We have developed a fix for the authentication errors affecting delivery to some Webhook destinations, and we are now actively deploying it across affected customers. The remediation restores the reset header values automatically, so impacted destinations will resume delivery as the rollout reaches them.

  6. resolved

    This incident has been resolved. If your workspace has Data Retention enabled, we are going to replay the events automatically.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). RudderStack profile · Recent RudderStack incidents · Markdown · JSON