Doppler: Background job delays impacting sync and email delivery

Impact: criticalStatus: resolvedStarted 2025-08-20 18:52 UTCResolved 2025-08-20 19:58 UTC (1.1 h)5 updatesSource: stspg.io

Summary

Doppler reported a critical background-job degradation from 2025-08-20 18:52 UTC to 2025-08-20 19:58 UTC, lasting 1 h 6 min. Integration syncs showed failed with "Out of sync", while email, activity log, and webhook delivery were delayed. A fix was implemented and all unprocessed jobs were processed by 19:38 UTC; the incident was resolved at 19:58 UTC.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Integrations - Twilio RotationTwilio RotationCodefreshIntegrations - CodefreshIntegrations - RenderRenderIntegrations - Laravel ForgeLaravel ForgeAWS User Key RotationIntegrations - AWS User Key RotationAWS Parameter StoreIntegrations - AWS Parameter StoreIntegrations - SplunkSplunkHerokuIntegrations - HerokuAzure Dev OpsIntegrations - Azure Dev OpsDashboard (dashboard.doppler.com)AWS MySQL RotationIntegrations - AWS MySQL RotationAWS IAM Dynamic SecretsIntegrations - AWS IAM Dynamic SecretsIntegrations - VercelVercelAWS Secrets ManagerIntegrations - AWS Secrets ManagerGCP Secret ManagerIntegrations - GCP Secret ManagerIntegrations - VercelVercelIntegrations - NetlifyNetlifyIntegrations - RailwayRailwayAzure App ServiceIntegrations - Azure App ServiceIntegrations - SendGrid RotationSendGrid RotationFly.ioIntegrations - Fly.ioCircleCIIntegrations - CircleCIAzure Key VaultIntegrations - Azure Key VaultGitHubIntegrations - GitHub

Timeline of updates

investigating2025-08-20 18:52 UTC
We're currently investigating an issue involving delays in background job processing that's impacting integration syncs, email delivery, activity log delivery, and webhook delivery (amongst other things). Impacted integration syncs will show as failed with "Out of sync". If expected emails aren't being received, it's likely related to this as well.
investigating2025-08-20 19:15 UTC
We are continuing to investigate the issue.
identified2025-08-20 19:36 UTC
The issue has been identified and a fix is being implemented.
monitoring2025-08-20 19:38 UTC
A fix has been implemented and we're monitoring the situation. All unprocessed jobs have now been processed.
resolved2025-08-20 19:58 UTC
This incident has been resolved.

Rewindex changes for this incident

Critical incident: Background job delays impacting sync and email delivery 2025-08-20 18:52 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Doppler profile · Recent Doppler incidents · Markdown · JSON