Fixing Missing Downstream Lineage When Using dbt and BigQuery Together – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Lineage Configuration — dbt + BigQuery Incremental Lineage

Issue

When running both a dbt ingestion and a BigQuery ingestion against the same tables, downstream lineage edges created by the dbt connector can appear to vanish after the BigQuery ingestion runs. The result is that navigating to an upstream dbt model in the DataHub Lineage Explorer shows zero downstream nodes, even though the dependency is correctly declared in the dbt manifest and the downstream dbt model correctly shows the BigQuery representation of the upstream table in its own lineage view. The root cause is a configuration mismatch: the dbt connector defaults to incremental_lineage: true, which adds edges without removing others, while the BigQuery connector defaults to incremental_lineage: false, which replaces the full lineage on every run — overwriting the edges that dbt created.

Error Messages

Upstream dbt URN lineage: 0 downstream nodes returned

You Might Be Asking

Why does the dbt model show no downstream dependencies in the Lineage Explorer, even though the downstream model shows the correct upstreams?
Why does lineage oscillate between runs — appearing correct sometimes and broken other times?
Why does querying the GraphQL API with separateSiblings: false return the expected downstream, but the UI shows nothing?
Can I configure the Lineage V3 UI to merge sibling edges from both the dbt and BigQuery URNs?

Solution

Understand the root cause.
When a dbt model depends on another dbt model that is materialized in BigQuery, the dbt connector stores the downstream edge on the BigQuery URN (because dbt lineage flows through the target platform). The dbt connector uses incremental_lineage: true by default, so it appends that edge without touching any existing edges. The BigQuery connector, however, defaults to incremental_lineage: false. Each time the BigQuery ingestion runs, it performs a full replacement of the table's lineage, removing the edge that dbt added. This causes the lineage to oscillate: correct after a dbt run, broken after the next BigQuery run.

Enable incremental_lineage on your BigQuery recipe.

Add the following setting to your BigQuery source configuration:

source:
  type: bigquery
  config:
    incremental_lineage: true
    # ... rest of your existing BigQuery config
    include_table_lineage: true
    include_usage_statistics: true
    include_tables: true
    include_views: true
    stateful_ingestion:
      enabled: true

With this setting enabled, the BigQuery ingestion will add to the existing lineage rather than replacing it, preserving the edges that the dbt connector creates.

Verify your dbt recipe does not need changes.

The dbt connector already defaults to incremental_lineage: true, so no change to your dbt recipe is required. A minimal valid dbt recipe for reference:

source:
  type: dbt
  config:
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    target_platform: bigquery
    env: PROD
    stateful_ingestion:
      enabled: true
    # incremental_lineage defaults to true — no change needed
sink:
  type: datahub-rest
  config:
    server: 'https://<your-instance>.acryl.io/gms'
    token: '<your-token>'

Re-run the dbt ingestion after updating the BigQuery recipe.
After saving the updated BigQuery recipe, trigger a manual dbt ingestion run to restore any edges that were previously removed. From that point forward, BigQuery ingestion runs will append to the dbt-created lineage rather than overwriting it.
Verify lineage in the UI.
Navigate to the upstream dbt model in the DataHub Lineage Explorer. It should now display the expected downstream dbt nodes without requiring users to switch to the BigQuery entity representation.

Additional Notes

Trade-off with incremental lineage: When incremental_lineage: true is set on the BigQuery connector, upstream edges that disappear from BigQuery query history are not automatically removed. If a table's actual sources change over time, stale lineage edges may persist until they are manually cleaned up or a full (non-incremental) ingestion is run.

UI behavior with separateSiblings: The Lineage V3 UI always sends separateSiblings: true when fetching lineage (hardcoded in useBulkEntityLineage.ts). This means the Explorer shows the dbt and BigQuery entities as separate nodes and does not merge sibling edges automatically. The UI reaches downstream nodes by following the dbt → BigQuery edge, which is why that edge being absent causes the graph to appear disconnected. This is a product limitation, not a configuration flag — fixing the incremental_lineage setting on the BigQuery recipe is the correct resolution. Using the GraphQL API directly with separateSiblings: false will return merged lineage, but this mode is limited to single-hop queries.

Version note: This behavior has been confirmed on acryl-datahub client version 1.3.x and GMS v2.3.x. The default values for incremental_lineage may vary across connector versions; always check the connector documentation for the version you are running.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation