BigQuery Ingestion Using Workload Identity Federation on Remote Executor in DataHub Cloud – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: BigQuery Authentication / Keyless Credentials

Issue

Users running DataHub Cloud who want to ingest metadata from BigQuery without long-lived service account credentials (private keys) may find that the guided ingestion form only exposes private-key fields and does not surface Workload Identity or Workload Identity Federation options. This leads to confusion about whether keyless authentication is supported in DataHub Cloud at all. In fact, keyless authentication is supported, but it requires a Remote Executor deployed inside your Google Cloud environment and configuration through the YAML recipe editor rather than the guided UI form.

You Might Be Asking

Can I use GKE Workload Identity or Workload Identity Federation for BigQuery ingestion in DataHub Cloud?
Why does the BigQuery ingestion form only show private key fields in DataHub Cloud?
How do I configure keyless BigQuery authentication using the YAML recipe editor in DataHub Cloud?
Do I need a Remote Executor to use Workload Identity with BigQuery in DataHub Cloud?

Solution

Keyless BigQuery authentication in DataHub Cloud requires two things: a Remote Executor running inside your Google Cloud environment, and configuration via the YAML recipe editor. Follow the steps below based on your preferred keyless method.

Ensure you have a Remote Executor deployed in your GKE cluster.
Managed (cloud-hosted) executors run outside your Google Cloud environment and have no GCP identity. Keyless methods rely on a workload having an identity that Google trusts, so they require an executor running within your environment. Deploy and register a Remote Executor in your GKE cluster if you have not already done so. See the Remote Executor documentation linked below.
Open the BigQuery ingestion source and switch to the YAML editor.
In DataHub Cloud, navigate to Ingestion > Sources, open your BigQuery source (or create a new one), and click the toggle to switch from the guided form view to the YAML / Recipe editor. The guided form exposes only the most common options (private key fields); the YAML editor gives access to all configuration options including keyless authentication.

Choose your keyless authentication method and configure the recipe accordingly.

Option A — GKE Workload Identity (simplest)

Bind the Remote Executor's Kubernetes service account to a Google service account that has the required BigQuery permissions. Then omit the credential block entirely from the recipe. The BigQuery connector falls back to Application Default Credentials (ADC) and automatically picks up the pod's workload identity.

source:
  type: bigquery
  config:
    project_id: <your-gcp-project-id>
    # No credential block needed — ADC picks up the pod's Workload Identity
    dataset_pattern:
      allow:
        - <your-dataset>

pipeline_name: bigquery_workload_identity

Option B — Workload Identity Federation

Set auth_type: workload_identity_federation and supply the external account configuration generated by gcloud iam workload-identity-pools create-cred-config. Use one of the three available fields depending on how you want to supply the config. Do not include a credential block when using this auth type — the validator will reject both together.

Using an inline JSON object (gcp_wif_configuration_json):

source:
  type: bigquery
  config:
    project_id: <your-gcp-project-id>
    auth_type: workload_identity_federation
    gcp_wif_configuration_json:
      type: external_account
      audience: //iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<pool-id>/providers/<provider-id>
      subject_token_type: urn:ietf:params:oauth:token-type:jwt
      token_url: https://sts.googleapis.com/v1/token
      credential_source:
        file: /var/run/secrets/tokens/gcp-ksa/token
      service_account_impersonation_url: https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<sa-name>@<your-gcp-project-id>.iam.gserviceaccount.com:generateAccessToken

pipeline_name: bigquery_wif

Alternatively, supply the config as a JSON string stored in a DataHub secret (gcp_wif_configuration_json_string):

source:
  type: bigquery
  config:
    project_id: <your-gcp-project-id>
    auth_type: workload_identity_federation
    gcp_wif_configuration_json_string: '{"type":"external_account","audience":"//iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<pool-id>/providers/<provider-id>","subject_token_type":"urn:ietf:params:oauth:token-type:jwt","token_url":"https://sts.googleapis.com/v1/token","credential_source":{"file":"/var/run/secrets/tokens/gcp-ksa/token"},"service_account_impersonation_url":"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<sa-name>@<your-gcp-project-id>.iam.gserviceaccount.com:generateAccessToken"}'

pipeline_name: bigquery_wif_string

Or supply a file path on the executor pod (gcp_wif_configuration):

source:
  type: bigquery
  config:
    project_id: <your-gcp-project-id>
    auth_type: workload_identity_federation
    gcp_wif_configuration: /path/to/external_account_config.json

pipeline_name: bigquery_wif_file

Select your Remote Executor for this ingestion source.
In the ingestion source settings, set the Executor field to your Remote Executor (the one running in your GKE cluster). This ensures the recipe runs in a pod that has access to the projected Kubernetes service account token or the WIF credential file referenced in the recipe.
Save and run the ingestion source.
Save the recipe and trigger a test run. Review the execution logs to confirm that authentication succeeds without a private key.

Additional Notes

The BigQuery ingestion UI form in DataHub Cloud currently surfaces only private-key credential fields. The YAML/recipe editor must be used for any keyless configuration. A future UI enhancement may add an authentication type selector to the guided form.
Workload Identity Federation support in the BigQuery connector was introduced in DataHub CLI version v2.2.2-cloud and later. Ensure your Remote Executor image is running a CLI version that includes this change.
Managed (cloud-hosted) executors cannot read token files from your Kubernetes cluster and therefore cannot use GKE Workload Identity or file-based Workload Identity Federation. A Remote Executor deployed inside your GCP environment is required for all keyless authentication flows.
When using auth_type: workload_identity_federation, you must not include a credential block in the same recipe. The configuration validator will reject the combination and the ingestion run will fail.
The projected Kubernetes service account token file path (e.g., /var/run/secrets/tokens/gcp-ksa/token) must be mounted into the executor pod. Verify your Remote Executor Helm chart or pod spec includes the correct volume and volume mount for the token.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation