Companies / Census / incidents

Census: Delays in Background Jobs

Affected components: Sync Engine

Timeline of updates

1. identified2025-10-20 13:43 UTC

   In our US region, we are seeing increased latency in non-sync background tasks like metadata refreshes and connection tests due to issues scaling up worker load to match demand, as a result of the continued downstream provider outage. We have identified the issue and are now mitigating the impact.

2. identified2025-10-20 16:07 UTC

   Non-sync background tasks are running normally. Email notifications are still delayed. We have detected syncs that are currently stuck in working state due to issues earlier in the day. We are working to unblock them.

3. identified2025-10-20 17:54 UTC

   We are seeing increases in Sync runtimes as we continue to experience issues adding new workers to our pool. We are continuing work to mitigate this issue.

4. monitoring2025-10-20 18:52 UTC

   We're now able to appropriately scale workers to meet demands. We are through our main backlogs (including mail sends), and Sync run latencies have improved substantially. We are still continuing to see transient errors throughout Sync runs, mainly dealing with issues hitting our Redis instances, but the frequency of these errors is continuing to decrease.

5. monitoring2025-10-20 20:51 UTC

   Our systems show sustained recovery and remain stable. We will continue to monitor for any further issues.

6. resolved2025-10-21 00:26 UTC

   Our system has been operating normally for the last few hours, and the aforementioned downstream provider has marked their incident as resolved. We are marking the sync engine as operational. Please contact support if you believe you are continuing to see issues.

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Census profile · Recent Census incidents · Markdown · JSON