Companies / Dagster / incidents
Dagster: Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error
Some Dagster asset runs failed with GraphQLStorageError errors as storage-engine timeouts increased during execution, affecting the API. The vendor rated the incident major; it ran from 2023-09-06 19:30 UTC to 2023-09-06 23:59 UTC, lasting 4 h 29 min. A fix was implemented and the incident was resolved.
Machine-written from the archived incident (Sol). Check the timeline before relying on it.
Affected components: API
Timeline of updates
- investigating
We have observed an increase in timeouts from our storage engine during run execution, resulting in some runs failing with an error like the following in the event log: dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: XXX)', 'locations': [{'line': 22, 'column': 13}], 'path': ['eventLogs', 'getEventRecords']}] We are currently investigating the issue.
- identified
The issue has been identified and we are in the process of testing a fix.
- monitoring
A fix has been implemented and we are monitoring the results.
- resolved
This incident has been resolved.
Rewindex changes for this incident
- Major incident: Some asset runs failing with a "GraphQLStorageError: Error in GraphQL Response" error 2023-09-06 19:30 UTC
Impact, status and times are the vendor’s own, as last archived. View on stspg.io (opens in new tab). Dagster profile · Recent Dagster incidents · Markdown · JSON