BigQuery Ingestion Failing Due to Missing bigquery.jobs.listAll Permission – DataHub Customer Support Portal

```html

Area: Ingestion Issues

Sub-Area: BigQuery Permissions & INFORMATION_SCHEMA Access

Issue

BigQuery ingestion in DataHub may fail with a 403 Access Denied error when the ingestion source is configured to query INFORMATION_SCHEMA.JOBS for usage statistics. This occurs when the service account used for ingestion does not have the bigquery.jobs.listAll permission at the project level. The error can appear across multiple regions (e.g., region-us, region-eu) for any project included in the ingestion configuration, causing the entire ingestion run to fail or produce incomplete metadata.

Error Messages

403 Access Denied: Table <project-id>:region-us.INFORMATION_SCHEMA.JOBS: User does not have permission to query table <project-id>:region-us.INFORMATION_SCHEMA.JOBS, or perhaps it does not exist.; reason: accessDenied
403 Access Denied: Table <project-id>:region-eu.INFORMATION_SCHEMA.JOBS: User does not have the required permissions ('bigquery.jobs.listAll' permission(s) at the project level) to query system entity <project-id>:region-eu.INFORMATION_SCHEMA.JOBS.; reason: accessDenied

You Might Be Asking

Why is my BigQuery ingestion returning a 403 error for INFORMATION_SCHEMA.JOBS?
What permissions does my BigQuery service account need for DataHub ingestion?
How do I grant bigquery.jobs.listAll to a service account?
Why does BigQuery ingestion fail in some regions but not others?
Can I skip usage ingestion for a specific BigQuery project to avoid this error?

Solution

There are two ways to resolve this issue depending on your requirements:

Option 1: Grant the Required Permission to the Service Account (Recommended)

The service account used for DataHub's BigQuery ingestion must have the bigquery.jobs.listAll permission at the project level for every project included in the ingestion configuration. This permission is included in the roles/bigquery.resourceAdmin role, or you can create a custom IAM role.

Identify all BigQuery projects included in your DataHub ingestion recipe (e.g., under project_ids or discovered via project_on_behalf).

For each project, grant the service account the necessary role using the gcloud CLI:

gcloud projects add-iam-policy-binding <your-project-id> \
  --member="serviceAccount:<your-service-account>@<your-project-id>.iam.gserviceaccount.com" \
  --role="roles/bigquery.resourceAdmin"

Alternatively, create a custom IAM role that includes only bigquery.jobs.listAll and bind it to the service account:

# Create a custom role with the required permission
gcloud iam roles create datahub_bq_usage_reader \
  --project=<your-project-id> \
  --title="DataHub BigQuery Usage Reader" \
  --permissions="bigquery.jobs.listAll"

# Bind the custom role to your service account
gcloud projects add-iam-policy-binding <your-project-id> \
  --member="serviceAccount:<your-service-account>@<your-project-id>.iam.gserviceaccount.com" \
  --role="projects/<your-project-id>/roles/datahub_bq_usage_reader"

Repeat for every project in every region that is included in the ingestion scope.
Re-run the BigQuery ingestion source in DataHub and confirm the 403 errors no longer appear.

Option 2: Exclude the Problematic Project from Ingestion

If granting permissions is not possible for a specific project, you can exclude it from the ingestion configuration using the project_ids or exclude_projects filter in your recipe.

Open your BigQuery ingestion recipe and locate the source configuration block.

Add the project to the exclusion list:

source:
  type: bigquery
  config:
    project_ids:
      - <project-id-with-permissions>
    # Remove or exclude the project lacking bigquery.jobs.listAll
    # project_ids should only list projects where the service account has full access

Alternatively, if you use project discovery, add an explicit deny pattern:

source:
  type: bigquery
  config:
    project_id_pattern:
      deny:
        - "^<project-id-without-permissions>$"

Save and re-run the ingestion. The excluded project will not be crawled for usage data.

Option 3: Disable Usage Ingestion Entirely

If usage statistics (INFORMATION_SCHEMA.JOBS queries) are not required, you can disable usage ingestion in the recipe:

Set the following flag in your BigQuery ingestion recipe:

source:
  type: bigquery
  config:
    include_usage_statistics: false

Save and re-run the ingestion. This will skip all INFORMATION_SCHEMA.JOBS queries and eliminate the permission errors.

Additional Notes

The bigquery.jobs.listAll permission is required at the project level, not the dataset or table level. Granting it at a lower scope will not resolve the error.
This error will appear independently per region (e.g., region-us, region-eu) for each project. Ensure permissions are consistent across all regions used by your BigQuery projects.
The INFORMATION_SCHEMA.JOBS table is queried specifically for usage statistics (query history, lineage from query logs). If your ingestion does not require usage data, Option 3 (disabling usage ingestion) is the lowest-effort resolution.
If your BigQuery ingestion was previously working and this error is new, check whether the service account's IAM bindings have changed, or whether new projects have been added to the ingestion scope that do not have the required permissions.
Debug mode enabled on an ingestion source can significantly increase log verbosity and memory usage, potentially causing out-of-memory (OOM) failures in executor environments before permission errors are even surfaced. Ensure debug logging is disabled for production ingestion runs.