Changes / Census / Sync execution halted
Census: Critical incident: Sync execution halted
Summary
Census’s critical Sync Engine outage paused all production syncs starting 2024-08-23 17:41 UTC; many syncs failed after partial restoration. Census ruled out duplicate-record concerns and deployed a fix, restoring expected success rates with backlog delays. The incident was resolved at 2024-08-23 19:15 UTC after 1 h 33 min.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Sync execution halted
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2024-08-23 17:41 UTC
Before: not present
After: 2024-08-23 19:15 UTC
Before: not present
After: Sync Engine
Before: not present
After: Production sync execution has been paused while we investigate an issue
Before: not present
After: We have identified an issue which could cause records to be synced multiple times to destinations. Since this can affect the correctness of syncs for certain destinations, we have paused all syncs proactively until the issue can be diagnosed and resolved. We are continuing to investigate and are in the act of rolling back to a previous version of the sync engine.
Before: not present
After: We have determined that the bug does not affect all syncs, nor does it cause records to be sent multiple times - this was a red herring.
A high percentage of syncs are still failing, but there is not a correctness issue, so we are reenabling the sync engine globally so that syncs that are not affected by the issue can return to operation.
Before: not present
After: We have identified the likely root cause and are deploying a fix at this time
Before: not present
After: Sync success rates have returned to expected levels. There is some delay in sync execution as we work through the backlog during the outage period. We are continuing to monitor
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Sync execution halted (this change) Census’s critical Sync Engine outage paused all production syncs starting 2024-08-23 17:41 UTC; many syncs failed after partial restoration. Census ruled out duplicate-record concerns and deployed a fix, restoring expected success rates with backlog delays. The incident was resolved at 2024-08-23 19:15 UTC after 1 h 33 min. | Major incident |