Changes / Doppler / Background job delays impacting sync and email delivery
Doppler: Critical incident: Background job delays impacting sync and email delivery
Summary
Doppler reported a critical background-job degradation from 2025-08-20 18:52 UTC to 2025-08-20 19:58 UTC, lasting 1 h 6 min. Integration syncs showed failed with "Out of sync", while email, activity log, and webhook delivery were delayed. A fix was implemented and all unprocessed jobs were processed by 19:38 UTC; the incident was resolved at 19:58 UTC.
Machine-written from the diff below (Sol; type: incident degradation). Check the diff before relying on it.
What changed
Before: not present
After: Background job delays impacting sync and email delivery
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2025-08-20 18:52 UTC
Before: not present
After: 2025-08-20 19:58 UTC
Before: not present
After: AWS IAM Dynamic Secrets, AWS MySQL Rotation, AWS Parameter Store, AWS Secrets Manager, AWS User Key Rotation, Azure App Service, Azure Dev Ops, Azure Key Vault, CircleCI, Codefresh, Dashboard (dashboard.doppler.com), Fly.io, GCP Secret Manager, GitHub, Heroku, Integrations - AWS IAM Dynamic Secrets, Integrations - AWS MySQL Rotation, Integrations - AWS Parameter Store, Integrations - AWS Secrets Manager, Integrations - AWS User Key Rotation, Integrations - Azure App Service, Integrations - Azure Dev Ops, Integrations - Azure Key Vault, Integrations - CircleCI, Integrations - Codefresh, Integrations - Fly.io, Integrations - GCP Secret Manager, Integrations - GitHub, Integrations - Heroku, Integrations - Laravel Forge, Integrations - Netlify, Integrations - Railway, Integrations - Render, Integrations - SendGrid Rotation, Integrations - Splunk, Integrations - Twilio Rotation, Integrations - Vercel, Laravel Forge, Netlify, Railway, Render, SendGrid Rotation, Splunk, Twilio Rotation, Vercel
Before: not present
After: We're currently investigating an issue involving delays in background job processing that's impacting integration syncs, email delivery, activity log delivery, and webhook delivery (amongst other things). Impacted integration syncs will show as failed with "Out of sync". If expected emails aren't being received, it's likely related to this as well.
Before: not present
After: We are continuing to investigate the issue.
Before: not present
After: The issue has been identified and a fix is being implemented.
Before: not present
After: A fix has been implemented and we're monitoring the situation. All unprocessed jobs have now been processed.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Background job delays impacting sync and email delivery (this change) Doppler reported a critical background-job degradation from 2025-08-20 18:52 UTC to 2025-08-20 19:58 UTC, lasting 1 h 6 min. Integration syncs showed failed with "Out of sync", while email, activity log, and webhook delivery were delayed. A fix was implemented and all unprocessed jobs were processed by 19:38 UTC; the incident was resolved at 19:58 UTC. | Major incident |