Area: Ingestion
Sub-Area: BigQuery Source
Issue
When using the BigQuery ingestion source with remote executors, the `qualifiedName` property was not being set on parent container entities (projects and datasets). This resulted in incomplete metadata hierarchies and made it difficult to properly navigate and organize BigQuery assets within DataHub.
Solution
The BigQuery connector has been updated to properly set the qualifiedName property for all container levels when using remote executors. This ensures consistent metadata structure across all BigQuery assets.
The fix is automatically applied when running ingestion with remote executors. No configuration changes are required:
Example for BigQuery. This can be adapted for other data sources/connectors.
source:
type: bigquery
config:
project_id: "your-gcp-project"
# The qualifiedName is now automatically populated
# for projects, datasets, and tables
Additional Notes
Fixed in version v0.3.11. This fix specifically affects remote executor deployments. Self-hosted ingestion may have already been handling this correctly. Rerun your BigQuery ingestion after upgrading to populate the qualifiedName for existing entities.
Related Documentation
Tags:
ingestion, bigquery, remote-executor, qualified-name, containers, metadata, bug-fix, v0.3.11