BigQuery Ingestion Permission Errors

BigQuery ingestion fails with permission errors when attempting to extract metadata, run profiling queries, or access specific datasets. This typically occurs when the service account lacks required IAM roles or when data access controls prevent metadata extraction.

Common Error Messages:

• Permission denied: bigquery.tables.get
• Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials
• User does not have bigquery.jobs.create permission
• 403 Forbidden: Access denied to resource

You Might Be Asking:

• What IAM roles does the DataHub service account need for BigQuery?
• How do I grant permissions for cross-project BigQuery ingestion?
• Why can't DataHub access my BigQuery tables?

1. Grant minimum required IAM roles to the service account:

Example for BigQuery. This can be adapted for other data sources/connectors.

# Basic metadata extraction
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:datahub@PROJECT.iam.gserviceaccount.com" \
  --role="roles/bigquery.metadataViewer"

# For usage and lineage (requires query job creation)
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:datahub@PROJECT.iam.gserviceaccount.com" \
  --role="roles/bigquery.jobUser"

# For profiling (requires data access)
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:datahub@PROJECT.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataViewer"

2. For cross-project ingestion, grant permissions on all projects:

Example for BigQuery. This can be adapted for other data sources/connectors.

source:
  type: bigquery
  config:
    project_ids: ["project-1", "project-2", "project-3"]
    # Service account needs permissions on all projects

3. Configure the credential file:

Example for BigQuery. This can be adapted for other data sources/connectors.

source:
  type: bigquery
  config:
    credential:
      project_id: "your-project"
      private_key_id: "key-id"
      private_key: "${GOOGLE_PRIVATE_KEY}"
      client_email: "datahub@project.iam.gserviceaccount.com"

4. For dataset-level access control:

Example for BigQuery. This can be adapted for other data sources/connectors.

# Grant access to specific datasets
bq show --format=prettyjson PROJECT:DATASET | \
  jq '.access += [{"role": "READER", "userByEmail": "datahub@project.iam.gserviceaccount.com"}]' | \
  bq update --source /dev/stdin PROJECT:DATASET

If using Workload Identity with GKE, ensure the Kubernetes service account is properly bound to the Google service account. Test permissions using bq query with the service account before debugging DataHub-specific issues.

• BigQuery Source Configuration
• GCP IAM Roles