Companies / Census / incidents
Census: Sync execution halted
Census’s critical Sync Engine outage paused all production syncs starting 2024-08-23 17:41 UTC; many syncs failed after partial restoration. Census ruled out duplicate-record concerns and deployed a fix, restoring expected success rates with backlog delays. The incident was resolved at 2024-08-23 19:15 UTC after 1 h 33 min.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: Sync Engine
Timeline of updates
- identified
Production sync execution has been paused while we investigate an issue
- identified
We have identified an issue which could cause records to be synced multiple times to destinations. Since this can affect the correctness of syncs for certain destinations, we have paused all syncs proactively until the issue can be diagnosed and resolved. We are continuing to investigate and are in the act of rolling back to a previous version of the sync engine.
- identified
We have determined that the bug does not affect all syncs, nor does it cause records to be sent multiple times - this was a red herring. A high percentage of syncs are still failing, but there is not a correctness issue, so we are reenabling the sync engine globally so that syncs that are not affected by the issue can return to operation.
- identified
We have identified the likely root cause and are deploying a fix at this time
- monitoring
Sync success rates have returned to expected levels. There is some delay in sync execution as we work through the backlog during the outage period. We are continuing to monitor
- resolved
This incident has been resolved.
Rewindex changes for this incident
- Critical incident: Sync execution halted 2024-08-23 17:41 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Census profile · Recent Census incidents · Markdown · JSON