Changes / Census / Sync Alerts missing as of last 2 days
Census: Major incident: Sync Alerts missing as of last 2 days
Summary
Census reported a major Sync Engine outage from 2024-09-20 20:18 UTC to 2024-09-20 22:55 UTC (2 h 37 min). Sync Alerts had been deleted from its database, preventing email alerts for qualifying sync runs over the prior two days. Alert data was restored, and Census said alerting should work again going forward.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Sync Alerts missing as of last 2 days
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-09-20 20:18 UTC
Before: not present
After: 2024-09-20 22:55 UTC
Before: not present
After: Sync Engine
Before: not present
After: Sync Alerts were deleted from our database in the last 2 days. The cause has been identified, and we are working on restoring the deleted alerts.
Before: not present
After: As part of this incident, any sync runs that should have triggered an alert in the last 2 days would not have resulted in an email alert. Once we restore the alert data, any sync runs that fail will trigger an alert, even though the sync may have been failing for the last 2 days. In very rare cases, we may email you at most twice about a failure as we perform the recovery process.
Before: not present
After: Sync Alert data has been restored. Sync alerting should work again for any syncs that trigger alerts moving forward.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Sync Alerts missing as of last 2 days (this change) Census reported a major Sync Engine outage from 2024-09-20 20:18 UTC to 2024-09-20 22:55 UTC (2 h 37 min). Sync Alerts had been deleted from its database, preventing email alerts for qualifying sync runs over the prior two days. Alert data was restored, and Census said alerting should work again going forward. | Major incident |