Companies/Census/Incidents

Census: Census sending duplicate / delayed status emails

Timeline of updates

1. resolved2023-02-10 05:21 UTC

   On 2023-02-09, the Census team identified an issue that was causing sync status emails to not be sent to a subset of customers. These emails report whether syncs succeed, fail, or other states based on customer configuration. We developed and deployed a fix for this issue at approximately 5:50 PM Pacific Time. At approximately 6:10 PM Pacific Time, we were alerted to the fact that Census was sending far more of these emails than usual. After a quick investigation, we discovered that the fix that had been applied had caused a “backfill” of status emails, and that some Census customers were receiving a large number of status emails for syncs that were days or weeks old. At approximately 6:20 PM Pacific, our devops team disabled email sending across the entire product to halt these unnecessary alerts. The product team worked to diagnose the root cause and also “undo” the backfill behavior. This fix was tested and deployed at approximately 9:15 PM Pacific Time, at which point email sending was reenabled for all customers. Customer Impact - A subset of Census customers received no emails for some or all of their sync statuses. We are currently investigating the extent of this monitoring outage and we will reach out to affected customers. - A subset of Census customers received a large flood of “backfilled” status emails - No Census customers received any status emails for any syncs that succeeded or failed between 6:20 PM Pacific and 9:15 PM Pacific on 2023-02-09 - The actual scheduling and execution of syncs was not affected - only the monitoring emails - Other email systems (password resets, invitations, weekly workspace summaries) were not affected

From Census’s published incident history: for incidents before Rewindex began polling this status page, only the final update text is available.

Impact, status and times are the vendor’s own, as last archived. View on status.getcensus.com (opens in new tab).

Census profileRecent Census incidentsAll outages on February 10, 2023Census in February 2023MarkdownJSON