BigQuery Ingestion: Fixing Project Discovery Inconsistencies with project_id_pattern – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: BigQuery Source Configuration & Project Discovery

Issue

When a BigQuery ingestion recipe relies solely on project_id_pattern for project discovery (without specifying project_ids or project_labels), DataHub calls the Google BigQuery projects.list API on each run to enumerate accessible projects. Because projects.list does not return a consistent snapshot — results can vary significantly from run to run due to changes in folder membership, org-level IAM bindings, or API pagination behavior — projects that have the BigQuery API disabled may appear unexpectedly on a given day and cause the entire ingestion run to fail with a 400 error. This failure is non-deterministic: the same disabled project may be absent from several consecutive runs and then suddenly appear, making the root cause difficult to diagnose.

Error Messages

400 BigQuery API has not been used in project <project-id> before or it is disabled.
BigQuery query log fetch failed

You Might Be Asking

Why does the number of projects discovered by DataHub change significantly from one day to the next with no changes on my side?
Why did a project with BigQuery API disabled suddenly cause my ingestion run to fail when it had never appeared in previous runs?
Can DataHub skip projects where the BigQuery API is not enabled instead of failing the entire run?
How can I make my BigQuery ingestion project discovery more stable and predictable?

Solution

Choose one of the following approaches in order of increasing stability. Each addresses the root cause at a different level.

Deny the specific problematic project (quickest short-term fix):
Add an explicit deny rule for the project whose BigQuery API is disabled. This prevents DataHub from attempting to ingest it even if projects.list returns it again.
```
source:
  type: bigquery
  config:
    project_id_pattern:
      allow:
        - ".*"          # keep your existing allow rules
      deny:
        - "^<disabled-project-id>$"   # replace with the actual project ID
```
Limitation: This only guards against the one known project. Other projects with BigQuery disabled could appear in the future and cause the same failure.
Tighten the allow pattern:
If your allow pattern is broad (for example .*-p$), consider narrowing it to match only the specific project ID prefixes or naming conventions that belong to your intended scope. This reduces the surface area of unexpected projects matching the pattern.
```
source:
  type: bigquery
  config:
    project_id_pattern:
      allow:
        - "^myorg-data-.*-p$"   # use a more specific naming prefix
```
Switch to project_labels for discovery:
Instead of relying on projects.list, use project_labels to discover projects via the Google Resource Manager search API. This targets only projects tagged with a specific label, making discovery more deterministic.
```
source:
  type: bigquery
  config:
    project_labels:
      - key: "datahub-ingest"
        value: "true"
```
Ensure the relevant GCP projects have this label applied before running ingestion.
Switch to an explicit project_ids list (most stable):
Eliminate auto-discovery entirely by specifying the exact list of project IDs to ingest. This guarantees that only known, intended projects are processed regardless of what projects.list returns on any given day.
```
source:
  type: bigquery
  config:
    project_ids:
      - "<your-project-id-1>"
      - "<your-project-id-2>"
      - "<your-project-id-3>"
```
Trade-off: You must keep this list up to date as GCP projects are added or removed in your environment.

To confirm whether project count variability is originating from Google's API, run the following command as the same service account used by your DataHub recipe on consecutive days and compare the output:

bq ls --projects --max_results=1000

If the project count changes without any IAM or recipe changes on your side, the variability is confirmed to originate from the projects.list API, not from DataHub.

Additional Notes

The projects.list API returns every GCP project where the service account has any IAM role, including roles inherited through folder or organization bindings. It does not filter by whether the BigQuery API is enabled on each project, and its paginated results are not a consistent snapshot.
As of the time of writing, DataHub does not automatically downgrade a "BigQuery API not enabled" error to a warning when projects are auto-discovered. When this error occurs — whether in fetch_query_log (queries extractor) or _process_project (schema extractor) — it is treated as a hard failure that marks the entire run as failed.
If projects.list itself encounters an error during the API call, DataHub logs the error and returns an empty list rather than retrying indefinitely. This can also cause unexpectedly low project counts on a given run. Check ingestion logs for the message Error getting projects if you observe a significantly smaller count than expected.
Using project_labels or project_ids is the recommended long-term approach for production pipelines where ingestion stability is critical.
If you believe unexpected projects are appearing due to IAM changes at the folder or organization level, review Cloud Audit Logs and IAM history for the service account used by your DataHub recipe.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation