Changes / Census / Sync runs failing

Census: Major incident: Sync runs failing

Summary

A subset of Census Sync Engine runs failed in a major-impact incident starting 2024-06-22 13:29 UTC and lasting 14 min, resolved at 2024-06-22 13:43 UTC. Census identified a storage issue affecting a core database and applied a fix, after which error rates decreased.

Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.

Major incidentLabel: outage (model (Jev), 86%)Changed Detected Source: stspg.io (opens in new tab)

What changed

AddedTitle#1
Before: not present
After: Sync runs failing
AddedImpact#2
Before: not present
After: major
AddedStatus#3
Before: not present
After: resolved
AddedStart time#4
Before: not present
After: 2024-06-22 13:29 UTC
AddedResolved time#5
Before: not present
After: 2024-06-22 13:43 UTC
AddedAffected components#6
Before: not present
After: Sync Engine
AddedUpdateinvestigating#7
Before: not present
After: A subset of sync runs are failing due to an infrastructure issue. We are investigating
AddedUpdatemonitoring#8
Before: not present
After: We have identified a storage issue affecting a core database and applied a fix - error rates are decreasing
AddedUpdateresolved#9
Before: not present
After: This incident has been resolved.

First version of this incident archived by Rewindex. View the live incident (opens in new tab).

History of this incident

When (UTC)ChangeKind
Major incident: Sync runs failing (this change)
A subset of Census Sync Engine runs failed in a major-impact incident starting 2024-06-22 13:29 UTC and lasting 14 min, resolved at 2024-06-22 13:43 UTC. Census identified a storage issue affecting a core database and applied a fix, after which error rates decreased.
Major incident

All Census changes → · Census profile · Markdown · JSON