BigQuery Ingestion - Incorrect Projects or Datasets

BigQuery ingestion is extracting datasets from incorrect or unintended projects, or filtering configuration is not working as expected to limit which projects/datasets are ingested into DataHub.

Note: For DataHub Cloud customers, infrastructure-level changes are managed by DataHub Support.

1. Configure project filtering:

source:
  type: bigquery
  config:
    project_id: "your-main-project"
    # Explicitly list allowed projects
    project_ids:
      - "analytics-prod"
      - "data-warehouse"
    # Or use patterns
    project_id_pattern:
      allow: ["prod-.*", "analytics-.*"]
      deny: ["test-.*", "dev-.*", ".*-sandbox"]

2. Verify service account permissions:

   • Ensure the service account only has BigQuery Data Viewer role on intended projects
   • Remove permissions from unwanted projects
   • Use project_ids list instead of patterns for precise control

3. Troubleshoot unexpected projects:

   • Check if service account was granted organization-level permissions
   • Review BigQuery audit logs to see which projects the service account accessed
   • Use include_table_lineage: true with caution as it scans query logs across accessible projects

4. Test with single project first:

   • Start with one project_id to confirm filtering works
   • Gradually add more projects using project_ids list
   • Verify in DataHub UI that only intended datasets appear

Test configurations in a development environment before applying to production. Always backup configurations before making changes.

• DataHub Documentation
• BigQuery Source

• 5692, 5643, 5606