Companies / dbt Labs / incidents
dbt Labs: Intermittent BigQuery job failures using Workload Identity Federation in EMEA
Affected components: 01JBCQVPBNMJ60EVAVX611QWT2
Timeline of updates
- investigating
We're investigating an issue with BigQuery authentication that is causing some job runs to fail intermittently. This is impacting jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. Affected runs fail with "The service is currently unavailable" (HTTP 503) while authenticating with Google Cloud, before any queries run. The team is working on a resolution and we will provide updates as soon as new information becomes available. In the meantime, rerunning a failed job or using Rerun from failure will usually succeed. Jobs using other BigQuery authentication methods, jobs on other data platforms, and dbt Studio are not impacted and should be running as expected.
- investigating
We're continuing to investigate intermittent failures for jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. Affected runs fail with "The service is currently unavailable" (HTTP 503) during the token exchange with Google Cloud. We haven't found any changes on our side, and the error appears to originate on Google Cloud's side. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
- investigating
We're continuing to investigate intermittent failures for jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
- investigating
We're continuing to investigate intermittent failures for jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
- monitoring
We have not observed any further occurrences and are monitoring to confirm the issue has fully cleared. dbt job runs have returned to their normal state and we are continuing to monitor.
- identified
We have identified an issue with intermittent failures for jobs that connect to BigQuery using Workload Identity Federation. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
- monitoring
The impacted component appears to be recovering, we will continue to monitor the performance. Please contact us at support@getdbt.com with any questions or concerns.
- resolved
The issue has been resolved, and all affected systems are now functioning normally. Please contact Support via email support@getdbt.com "support@getdbt.com": http://support@getdbt.com if you continue to experience delays and are unsure of the root cause.
Impact, status and times are the vendor’s own, as last archived. View on status.getdbt.com (opens in new tab). dbt Labs profile · Recent dbt Labs incidents · Markdown · JSON