Changes / Census / Census unavailable. 503 Error page from the Web.

Census: Major incident: Census unavailable. 503 Error page from the Web.

Summary

Census reported a major outage affecting the Sync Management UI and Sync Engine, with a 503 error page, from 2025-01-07 20:53 UTC to 2025-01-07 21:28 UTC (35 min). An infrastructure upgrade inadvertently removed services; all services recovered, and queued syncs began starting at 2025-01-07 21:18 UTC.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 100%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Census unavailable. 503 Error page from the Web.
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2025-01-07 20:53 UTC
AddedResolved time#5
Before: not present
After: 2025-01-07 21:28 UTC
AddedAffected components#6
Before: not present
After: Census Sync Management UI, Sync Engine
AddedUpdateinvestigating#7
Before: not present
After: We are investigating the issue and will update ASAP.
AddedUpdateinvestigating#8
Before: not present
After: The root cause has been identified and we are working to resolve.
AddedUpdateidentified#9
Before: not present
After: The Census UI is now back up. Our sync engine is in the process of coming back up.
AddedUpdatemonitoring#10
Before: not present
After: All services are operational. Any queued syncs are starting up now.
AddedUpdateresolved#11
Before: not present
After: Everything is now 100% operational. The problem came from an upgrade of our infrastructure that inadvertently removed services. We are planning now to make sure this dependency problem won't be at risk in future upgrades.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Census unavailable. 503 Error page from the Web. (this change)
Census reported a major outage affecting the Sync Management UI and Sync Engine, with a 503 error page, from 2025-01-07 20:53 UTC to 2025-01-07 21:28 UTC (35 min). An infrastructure upgrade inadvertently removed services; all services recovered, and queued syncs began starting at 2025-01-07 21:18 UTC.
Major incident

All Census changes → · Census profile · Markdown · JSON