BigQuery Connector: Usage Config Fields and Time Window Control Behavior – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: BigQuery Source Configuration

Issue

When configuring BigQuery ingestion, users may set time window parameters such as start_time and end_time under the nested usage: block of their recipe, expecting those values to control the range of queries and usage data that gets ingested. In practice, these nested usage.* time fields are ignored by both the default queries-v2 path (use_queries_v2: true) and the legacy audit log path. The connector always reads time window parameters from the top-level configuration. Additionally, several other fields within BigQueryUsageConfig are silently ignored when running the default queries-v2 extraction path, leading to unexpected behavior and making it difficult to understand which configuration values are actually effective.

Error Messages

No error is raised — dead config fields are silently ignored and ingestion completes with incorrect time window or missing behavior.

You Might Be Asking

Why is my BigQuery usage ingestion only returning data from the last day even though I set usage.start_time to a longer lookback window?
Which fields under usage: in my BigQuery recipe are actually respected?
Does usage.format_sql_queries work with the default BigQuery connector settings?
Why does setting usage.bucket_duration have no effect on my ingestion output?

Solution

The root cause is that BigQueryUsageConfig inherits fields from BaseTimeWindowConfig and BaseUsageConfig, but the connector internally reads time window values from the top-level BigQueryV2Config object — not from the nested usage sub-config. This affects both the queries-v2 path (default since use_queries_v2 was made True by default) and the legacy audit log path.

Move time window fields to the top level of your recipe config. Do not set usage.start_time, usage.end_time, usage.bucket_duration, or usage.max_query_duration — these are never read by either code path. Set them at the top level instead:

source:
  type: bigquery
  config:
    start_time: "-30d"       # Top-level — controls usage AND lineage time window
    end_time: "0d"           # Top-level
    bucket_duration: "DAY"   # Top-level
    project_ids:
      - <your-gcp-project-id>
    usage:
      top_n_queries: 50                  # Respected in both paths
      include_operational_stats: true    # Respected in both paths
      user_email_pattern:                # Respected in queries-v2 path only
        allow:
          - ".*@<your-domain>.com"

Understand which usage.* fields are effective in each extraction path. The table below summarizes the behavior of common BigQueryUsageConfig fields:

Field	Respected in queries-v2 (default)	Respected in legacy path	Notes
`usage.start_time`	❌ No	❌ No	Use top-level `start_time`
`usage.end_time`	❌ No	❌ No	Use top-level `end_time`
`usage.bucket_duration`	❌ No	❌ No	Use top-level `bucket_duration`
`usage.max_query_duration`	❌ No	❌ No	Use top-level `max_query_duration`
`usage.top_n_queries`	✅ Yes	✅ Yes	Works in both paths
`usage.include_operational_stats`	✅ Yes	✅ Yes	Works in both paths
`usage.user_email_pattern`	✅ Yes	❌ No	Only effective in queries-v2
`usage.format_sql_queries`	❌ No	✅ Yes	Hardcoded `False` in queries-v2
`usage.include_read_operational_stats`	❌ No	✅ Yes	Not read by queries-v2
`usage.apply_view_usage_to_tables`	❌ No	✅ Yes	Not read by queries-v2
`usage.include_top_n_queries`	❌ No*	✅ Yes	*Default matches, so no observable effect
`usage.queries_character_limit`	❌ No*	✅ Yes	*Default matches, so no observable effect

If you require format_sql_queries, include_read_operational_stats, or apply_view_usage_to_tables, you can fall back to the legacy extraction path by explicitly setting use_queries_v2: false. Be aware that the legacy path provides lower-quality lineage and query extraction compared to the queries-v2 path:
```
source:
  type: bigquery
  config:
    use_queries_v2: false    # Reverts to legacy audit log extractor
    start_time: "-30d"
    end_time: "0d"
    usage:
      format_sql_queries: true
      include_read_operational_stats: true
      apply_view_usage_to_tables: true
```
Upgrade to CLI v1.6.0.11 or later. A fix was merged in pull request datahub-project/datahub#18133 that addresses the documentation and wiring of these fields. Upgrading to CLI v1.6.0.11 or later will include these corrections:
```
# Check your current DataHub CLI version
datahub version

# Upgrade to the latest version
pip install --upgrade acryl-datahub
```

Additional Notes

The silent mismatch between usage.* field names and the actual fields read by the connector was introduced as a documentation and configuration clarity gap when use_queries_v2 was made the default. Fields such as format_sql_queries, include_read_operational_stats, and apply_view_usage_to_tables previously functioned correctly under the legacy path (when use_queries_v2 defaulted to false); their loss of effect is a behavioral regression for users who did not explicitly pin use_queries_v2. The fields usage.start_time and usage.end_time were never correctly wired in either path — this is a longstanding documentation issue predating the queries-v2 default change. The fix in CLI v1.6.0.11 addresses these discrepancies. If you are on a managed DataHub Cloud deployment, consult with your DataHub support contact regarding when the fix will be available in your environment.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation