Changes / Census / Partial Sync Outage
Census: Major incident: Partial Sync Outage
Summary
Census reported a major Sync Engine outage starting 2024-09-11 19:00 UTC, lasting 5 h 5 min and resolved at 2024-09-12 00:06 UTC. Empty unload batches could fail with file or Parquet conversion errors; after a fix, some syncs encountered function errors, primarily advanced sync engine mirror syncs with no deletes.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Partial Sync Outage
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2024-09-11 19:00 UTC
Before: not present
After: 2024-09-12 00:06 UTC
Before: not present
After: Sync Engine
Before: not present
After: Syncs empty unload batches have a chance of failing with either a "IO Error: No files found that match the pattern IGNORED_EMPTY_FILE" or a "Parquet Conversion Error".
The issue has been identified and a fix is being released.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: A subset of the previously affected syncs are now hitting "No function matches the given name and argument types" errors. It's primarily limited to advanced sync engine mirror syncs with no deletes. Investigation continues.
Before: not present
After: The issue has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Partial Sync Outage (this change) Census reported a major Sync Engine outage starting 2024-09-11 19:00 UTC, lasting 5 h 5 min and resolved at 2024-09-12 00:06 UTC. Empty unload batches could fail with file or Parquet conversion errors; after a fix, some syncs encountered function errors, primarily advanced sync engine mirror syncs with no deletes. | Major incident |