Axiom: Elevated query error rates
Affected components: Querying
Timeline of updates
- investigating
We are investigating elevated query error rates and increased latency affecting some customers. Our engineering team is actively working to identify the cause and restore normal operation.
We will provide further updates as more information becomes available.
- resolved
A portion of queries experienced increased latency and elevated error rates which was caused by an interaction between our service mesh and internal cache services. Following a deployment, some query workers continued attempting to connect to outdated cache instance addresses, even after the replacement instances had become healthy. This resulted in query timeouts and failures.
We identified the underlying issue and mitigated it by adjusting the networking configuration of our query workers. Normal operation was restored by 12:07 UTC, and query error rates have returned to their expected levels.
We are reviewing the incident and implementing measures to prevent similar issues in the future.
Archive proof
Rewindex commits each version of an incident it captures to a daily checkpoint, chains it to the day before, and has the checkpoint timestamped by FreeTSA and DigiCert.
How the archive is checked, and what it cannot prove.
Rewindex captured this after the last checkpoint; the next run () commits it.
Impact, status and times are the vendor’s own, as last archived. View on status.axiom.co (opens in new tab).