Changes / Census / Census Full Outage
Census: Critical incident: Census Full Outage
Summary
A critical Census outage affected Census Sync Management UI and Sync Engine from 2026-01-05 15:17 UTC to 2026-01-05 20:24 UTC (5 h 7 min). Syncs could not run, and website access was disrupted. Census attributed the outage to an indexing change and said rebuilding previous indexes delayed recovery.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Census Full Outage
Before: not present
After: critical
Before: not present
After: resolved
Before: not present
After: 2026-01-05 15:17 UTC
Before: not present
After: 2026-01-05 20:24 UTC
Before: not present
After: Census Sync Management UI, Sync Engine
Before: not present
After: Census is currently having issues running Syncs and rendering the Website. Engineers have identified the issue and are remediating.
Before: not present
After: We have identified and triaged the affected system and systems are coming back up now. Web is operational. We will see Sync delays as we scale back up.
Before: not present
After: We are still seeing degraded performance and in some cases failing syncs. We are continuing efforts to mitigate as the system repairs.
Before: not present
After: Syncs are currently not operational. Scheduled syncs should still be queuing up but they won't start running at this time. We will provide an update once workers come back up.
Before: not present
After: Syncs are now operational. Users may be experiencing issues loading Sync Configuration and Sync Run details on the web.
Before: not present
After: The web is now operating normally, along with Sync Runs. We are monitoring the system for overall health.
Before: not present
After: This incident has been resolved. Syncs have been running normally as of approximately 1:00pm EST, and the web has been running at normal latencies as of 2:00pm EST.
The issue was caused by an indexing change on our side that impacted our ability to run syncs reliably. Recovery was delayed because we needed to rebuild the previous indexes. To avoid any risk of data loss, we chose a safer recovery approach that allowed the system to remain temporarily degraded while repairs were being completed.
To restore syncs as quickly as possible, we prioritized sync execution and applied targeted fixes to improve performance. During this time, access to the UI was temporarily down.
First version of this incident archived by Rewindex. View the live incident.
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Critical incident: Census Full Outage (this change) A critical Census outage affected Census Sync Management UI and Sync Engine from 2026-01-05 15:17 UTC to 2026-01-05 20:24 UTC (5 h 7 min). Syncs could not run, and website access was disrupted. Census attributed the outage to an indexing change and said rebuilding previous indexes delayed recovery. | Major incident |