Resolving Incomplete or Broken Lineage When Using dbt with Snowflake in DataHub – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt + Snowflake Lineage Configuration

Issue

When using the dbt and Snowflake connectors together in DataHub, lineage can appear broken, dangling, or inconsistent. Common symptoms include lineage edges that terminate at dbt source nodes instead of resolving to the underlying Snowflake tables, upstream entities appearing to be missing after a Snowflake ingestion run, unexpected entities being created despite entities_enabled.sources: false, and schema deny-list patterns failing to exclude the correct schemas. These problems typically stem from a combination of recipe configuration flags that interact in non-obvious ways — specifically skip_sources_in_lineage, skip_missing_upstreams_in_lineage, the deprecated include_view_lineage flag, and incorrectly written schema_pattern.deny rules.

Error Messages

Lineage ends at dbt node; downstream Snowflake entity not connected
Open-ended / dangling lineage edges visible in the Lineage Explorer
schema_pattern.deny rules have no effect on filtered schemas

You Might Be Asking

Why does lineage look complete on the Lineage tab but broken on the Summary tab?
Why does running Snowflake ingestion break lineage that was working after a dbt run?
How do I prevent DataHub from creating lineage edges to Snowflake tables that no longer exist at the declared path?
Why is include_view_lineage: false not suppressing view-derived lineage?
Why are my schema_pattern.deny rules not excluding the schemas I specified?
Does Snowflake lineage include edges from user SELECT queries?

Solution

1. Understand the Summary Tab vs. Lineage Tab Difference

The Summary tab's lineage panel is a compact preview widget. It hard-caps at two upstream and two downstream entities per hop and renders whichever two happen to load first — it does not select the most important ones. It is intentionally lightweight and is meant only as a quick visual indicator that lineage exists. The Lineage tab is the authoritative, complete lineage explorer with no entity-per-hop cap. Always use the Lineage tab for reliable lineage investigation.

2. Understand How `skip_sources_in_lineage` and `entities_enabled.sources` Interact with Snowflake Lineage

When both flags below are set in the dbt recipe:

entities_enabled:
  sources: false
skip_sources_in_lineage: true

The dbt connector does not create a dedicated dbt-platform node for assets declared as dbt sources. Instead, it attempts to point lineage edges directly at the corresponding Snowflake entity. This works when the Snowflake entity's URN exactly matches the path declared in your sources.yml. However, if the underlying Snowflake object has been renamed, moved to a different schema, or replaced by a different object type (e.g., a Dynamic Table in a different schema), the edge will dead-end and produce broken lineage with no fallback node to anchor it.

Note: entities_enabled.sources: false only affects the dbt connector. It has no effect on the Snowflake connector. If a table is visible to the Snowflake connector, that connector will create or update the Snowflake entity on every run regardless of any dbt setting.

3. Add `skip_missing_upstreams_in_lineage: true` to Prevent Dangling Edges

To prevent DataHub from writing lineage edges to Snowflake entities that do not yet exist in DataHub's metadata store, add the following flag to your dbt recipe:

entities_enabled:
  sources: false
skip_sources_in_lineage: true
skip_missing_upstreams_in_lineage: true

Before writing an upstream edge, DataHub checks whether the target URN already exists as an entity. If it does not, the edge is silently dropped — no dangling lineage, no phantom entity. This gives you the behavior: "if the upstream cannot be resolved, show no lineage rather than incorrect lineage."

Important ordering caveat: The existence check runs against whatever is currently in DataHub, not directly against Snowflake. If the dbt ingestion runs before the Snowflake connector has ever ingested a given table, that edge will be dropped until the next dbt run after Snowflake has caught up. Always run Snowflake ingestion before dbt ingestion in your pipeline ordering.

Root-cause caveat: If a source path in your sources.yml is genuinely incorrect (e.g., referencing the wrong schema), skip_missing_upstreams_in_lineage will suppress the bad edge but will not fix the underlying declaration. Correct the sources.yml path to ensure accurate lineage once the Snowflake entity exists at the declared location.

4. Understand That `include_view_lineage: false` Is a Deprecated No-Op

include_view_lineage was removed in a past DataHub release. Setting it in your recipe has no effect — the field is stripped before it reaches the Snowflake connector's active configuration. The mechanism that computes lineage from view SQL definitions is currently hardcoded on. There is no configuration flag available to disable view-derived lineage while still ingesting view entities. If you want to remove view entities entirely, you can use include_views: false, but this removes the view entities themselves, not just their lineage edges.

Additionally, include_table_lineage: false gates only one specific mechanism: lineage computed from Snowflake's query history. It does not suppress view lineage or Dynamic Table lineage (which Snowflake derives natively from a dynamic table's declared inputs). Those mechanisms remain active regardless of this flag.

Regarding what Snowflake query-history lineage covers: only write-producing queries generate lineage edges (CREATE TABLE AS SELECT, INSERT INTO, MERGE, COPY INTO, etc.). Plain read-only SELECT queries are explicitly excluded. No additional configuration is needed to restrict lineage to write operations.

5. Fix `schema_pattern.deny` Rules to Correctly Exclude Schemas

If your schema_pattern.deny rules are written as fully qualified database.schema patterns but are not taking effect, you must also set match_fully_qualified_names: true. Without this flag, the pattern matcher compares against the schema name alone. Additionally, ensure regex special characters (such as .) are properly escaped.

Correct configuration example:

match_fully_qualified_names: true
schema_pattern:
  deny:
    - 'your_analytics_db\.usr_.*'
    - 'your_analytics_db\.your_schema_prefix\..*'

Replace your_analytics_db, usr_.*, and your_schema_prefix with the actual database name and schema patterns you want to exclude.

6. Complete Reference: Recommended dbt + Snowflake Recipe Flags

The following is a consolidated view of the key flags and their interactions:

# --- dbt recipe ---
entities_enabled:
  sources: false              # Do not create dbt-platform nodes for declared sources.
                              # dbt lineage will point directly at Snowflake entities.
skip_sources_in_lineage: true # Omit the intermediate dbt source node from lineage edges.
skip_missing_upstreams_in_lineage: true
                              # Drop any upstream edge whose target URN does not yet
                              # exist in DataHub. Prevents dangling/open-ended lineage.

# --- Snowflake recipe ---
include_table_lineage: false  # Disables lineage derived from Snowflake query history.
                              # Does NOT disable view-derived or Dynamic Table lineage.
# include_view_lineage: false # DEPRECATED — has no effect. Do not rely on this flag.

match_fully_qualified_names: true
schema_pattern:
  deny:
    - 'your_database\.your_schema_pattern_.*'

7. Cleaning Up Stale Lineage Edges

The stateful_ingestion.fail_safe_threshold setting soft-deletes entities that no longer appear in a source, but it requires two successful ingestion runs to establish state before soft-deletion takes effect on the first run. Stale lineage edges that remain after a source object has moved are only removed as a side effect of the source entity itself being soft-deleted. To programmatically remove individual lineage edges, use the DataHub Python SDK or the GraphQL API to delete specific upstream relationships by URN. Deprecating an entity in DataHub does not remove its lineage edges — lineage persists. Use soft-deletion if you want an entity to no longer appear in lineage graphs.

Additional Notes

include_view_lineage was removed from the Snowflake connector in a DataHub release prior to mid-2025. Check your DataHub version's changelog to confirm when this change took effect for your deployment.
dbt compiles source() references against your sources.yml declarations, not against a live warehouse check. A dbt run can compile successfully and reference a source that has since been renamed or moved in Snowflake. DataHub's dbt connector inherits this behavior — it does not independently validate that declared source paths resolve to real Snowflake objects.
Dynamic Table lineage in Snowflake is always derived from the dynamic table's native input declarations, independent of include_table_lineage or include_view_lineage settings.
Ingestion run ordering matters: run Snowflake ingestion before dbt ingestion to ensure upstream Snowflake entities exist in DataHub before dbt attempts to write lineage edges pointing to them.