Changes / Dagster / Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error
Dagster: Major incident: Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error
Summary
Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved.
Machine-written from the diff below (Sol; type: incident outage). Check the diff before relying on it.
What changed
Before: not present
After: Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error
Before: not present
After: major
Before: not present
After: resolved
Before: not present
After: 2023-09-06 19:30 UTC
Before: not present
After: 2023-09-06 23:59 UTC
Before: not present
After: API
Before: not present
After: We have observed an increase in timeouts from our storage engine during run execution, resulting in some runs failing with an error like the following in the event log:
dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: XXX)', 'locations': [{'line': 22, 'column': 13}], 'path': ['eventLogs', 'getEventRecords']}]
We are currently investigating the issue.
Before: not present
After: The issue has been identified and we are in the process of testing a fix.
Before: not present
After: A fix has been implemented and we are monitoring the results.
Before: not present
After: This incident has been resolved.
First version of this incident archived by Rewindex. View the live incident (opens in new tab).
History of this incident
| When (UTC) | Change | Kind |
|---|---|---|
| Major incident: Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error (this change) Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved. | Major incident |