Companies / Salesforce / incidents
Salesforce: Feature Perf Degradation: Analytics, Core Service
Affected components: AnalyticsCore Service78 instances
Timeline of updates
- update
We're investigating reports that data synchronization jobs for CRM Analytics (CRMA) are not processing as expected. As a result, a subset of customers may experience failures or delays in CRMA data sync operations and recipe jobs, which may appear stuck in the queue or fail to complete. We have identified a recent deployment as the potential trigger for the issue. We are performing a rollback on one of the impacted instances in an effort to resolve the issue. We'll provide an update in 30 minutes or sooner if additional information becomes available.
- update
We have completed the rollback on a test instance; however, post-rollback validation shows that jobs are still not progressing, indicating that this action did not resolve the underlying issue. We are now analysing system metrics and job-processing telemetry. Early signals highlight an increase in “workers are unhealthy or fail to accept job” errors, which aligns with the observed job failures. The team is preparing an additional rollback to an earlier version. We'll provide an update in 30 minutes or sooner if additional information becomes available.
- update
Upon further investigation, we have determined that the start time of impact is different than initially understood. We have now revised the start time of the Trust post to more accurately reflect the time customers may have begun to experience impact. We apologize for any confusion caused by this.
- update
Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.
- update
Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.
- update
We are continuing to monitor the progress of the rolling restarts across the impacted instances. We expect to complete this within the next 30 minutes. Early results show improvement, logs confirm an increasing number of jobs completing successfully, and validation workloads are now running to completion on the restarted worker nodes. We'll provide an update in 30 minutes or sooner if additional information becomes available.
- update
We have completed the rolling restart of all worker nodes and are actively monitoring to ensure the service continues to recover as expected. Additionally, we are monitoring the volume of the queued jobs, and they are gradually reducing to a healthier level. customers may continue to see an impact as the job backlog clears. Each customer org also has built-in concurrency thresholds based on licensing. Due to these limitations, customers may observe jobs remaining in a queued state even when no jobs appear to be actively running. This is expected behaviour and will continue for several hours as queued jobs are processed in order. Most job types, such as recipes and dataflows, should clear quickly once workers are healthy. However, some of these jobs depend on Data Sync completing first, which may extend their processing time. Customers do not need to cancel or restart any jobs; the system will automatically execute them as capacity becomes available. During this monitoring period, some scheduled Data Sync runs may temporarily fail with “Job already in queue” errors. These errors will not impact the jobs that are queued.. Customers may optionally unschedule affected connections to prevent repeated scheduling attempts, but this is not required for stability or recovery. We will continue to monitor worker health and job throughput for the next few hours until all metrics consistently reflect normal, healthy performance.
- update
Recent mitigation efforts, including the addition of two worker PODs and a rolling restart, have helped stabilize processing; however, the incident trigger is still being investigated, and the issue continues to have an impact and exhibit signs of regression. We are monitoring the volume of queued jobs and have discovered that some jobs are stuck in processing, consuming slots. As of 16:34 UTC, a second rolling restart has been started. This is expected to be completed in 30 to 45 minutes. We will provide an update in 60 minutes or sooner if additional information becomes available.
- update
The volume of queued jobs continues to accumulate, and we are implementing corrective actions to prevent further delays. A second set of rolling restarts is in progress to address any jobs that are currently stuck. We will continue to evaluate whether these actions work to resolve the issue. We will provide an update in 60 minutes or sooner if additional information becomes available.
- update
After monitoring performance and completing health checks following the rolling restarts, the team has confirmed that services have been restored as of 18:27 UTC. A small number of jobs may remain stuck in a running or queued state without making progress. Customers may need to manually cancel and restart these jobs. We will perform a full investigation of the incident, confirming the technical trigger, the underlying cause, and preventive action to avoid a repeat in the future. We apologize for how you and your business may have been affected by this incident.
Impact, status and times are the vendor’s own, as last archived. View on status.salesforce.com (opens in new tab). Salesforce profile · Recent Salesforce incidents · Markdown · JSON