Census: Partial Sync Outage

Impact: majorStatus: resolvedStarted 2024-09-11 19:00 UTCResolved 2024-09-12 00:06 UTC (5.1 h)4 updatesSource: stspg.io

Summary

Census reported a major Sync Engine outage starting 2024-09-11 19:00 UTC, lasting 5 h 5 min and resolved at 2024-09-12 00:06 UTC. Empty unload batches could fail with file or Parquet conversion errors; after a fix, some syncs encountered function errors, primarily advanced sync engine mirror syncs with no deletes.

Machine-written from the archived incident (Sol). Check the timeline before relying on it.

Affected components: Sync Engine

Timeline of updates

identified2024-09-11 19:00 UTC
Syncs empty unload batches have a chance of failing with either a "IO Error: No files found that match the pattern IGNORED_EMPTY_FILE" or a "Parquet Conversion Error". The issue has been identified and a fix is being released.
monitoring2024-09-11 19:23 UTC
A fix has been implemented and we are monitoring the results.
investigating2024-09-11 20:50 UTC
A subset of the previously affected syncs are now hitting "No function matches the given name and argument types" errors. It's primarily limited to advanced sync engine mirror syncs with no deletes. Investigation continues.
resolved2024-09-12 00:06 UTC
The issue has been resolved.

Rewindex changes for this incident

Major incident: Partial Sync Outage 2024-09-11 19:00 UTC

Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Census profile · Recent Census incidents · Markdown · JSON