Fixing Duplicate dbt URNs Caused by URN Casing Changes Breaking Sibling Lineage – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt Ingestion / URN Management / Sibling Lineage

Issue

When the convert_urns_to_lowercase setting is changed in a dbt ingestion recipe (for example, switching from false to true), DataHub generates new lowercase URNs for all dbt assets while the old uppercase URNs remain in the catalog as stale, orphaned entities. This results in duplicate dbt dataset entries — one with uppercase identifiers and one with lowercase — where the sibling relationship between the dbt asset and its warehouse counterpart (e.g., Snowflake) is still pointing to the old uppercase URN. Because the downstream lineage (such as dbt exposures or downstream consumers) is attached to the newer, correct lowercase URN rather than the stale uppercase one, the combined sibling view displays incomplete or missing lineage. Users navigating the default merged sibling view see no downstream lineage, and must manually switch to the individual warehouse component view to find it, which is not intuitive.

Error Messages

No downstream lineage visible on the sibling (merged) view of a dbt + warehouse dataset pair
Two dbt URNs exist for the same model, differing only in casing (e.g., urn:li:dataset:(urn:li:dataPlatform:dbt,MY_SCHEMA.MY_MODEL,PROD) vs urn:li:dataset:(urn:li:dataPlatform:dbt,my_schema.my_model,PROD))

You Might Be Asking

Why does the sibling lineage view show no downstream assets, but clicking into the individual dbt or warehouse component reveals full lineage?
Why do I have two dbt dataset entries for the same model that differ only in letter casing?
How do I remove stale uppercase dbt URNs after changing convert_urns_to_lowercase to true?
How do I fix broken sibling relationships after cleaning up duplicate URNs?
Will enabling stateful ingestion automatically remove stale dbt assets?

Solution

There are two complementary steps: (1) prevent stale duplicate URNs from accumulating in the future by enabling stateful ingestion, and (2) manually remediate any duplicate URNs and broken sibling relationships that already exist.

Step 1: Enable Stateful Ingestion on the dbt Recipe (Preventive)

Add the stateful_ingestion block to your dbt ingestion recipe. After two successful ingestion runs, any dbt asset that no longer exists in the dbt source will be soft-deleted. Soft-deleted assets are hidden from search and are permanently hard-deleted after 10 days.

source:
  type: dbt
  config:
    manifest_path: target/manifest.json
    catalog_path: target/catalog.json
    target_platform: snowflake
    stateful_ingestion:
      enabled: true

sink:
  type: datahub-rest
  config:
    server: https://<your-instance>.acryl.io/gms
    token: <your-datahub-token>

Once this is in place, stale URNs (including those generated by a prior casing configuration) will be cleaned up automatically over subsequent ingestion cycles.

Step 2: Identify Duplicate URNs

If you suspect duplicates exist, search for the affected dbt model in the DataHub UI. Look for two entries representing the same model — one with uppercase characters in the dataset identifier portion of the URN and one with lowercase. You can confirm by comparing their last-ingested timestamps; the stale uppercase URN will typically have an older sync date.

Example of a duplicate URN pair:

# Stale (uppercase) — old URN, broken sibling pointer
urn:li:dataset:(urn:li:dataPlatform:dbt,MY_SCHEMA.RELEASE.my_model_name,PROD)

# Current (lowercase) — correct URN with active lineage
urn:li:dataset:(urn:li:dataPlatform:dbt,my_schema.release.my_model_name,PROD)

Step 3: Reassign Sibling Relationships to the Correct URN

Before deleting the stale uppercase URN, you must reassign the sibling relationship. Sibling relationships in DataHub are bidirectional and must be updated on both sides. Use the DataHub REST API or Python SDK to patch the sibling aspect on both the warehouse asset and the correct lowercase dbt asset.

Using the DataHub Python SDK:

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import SiblingsClass

emitter = DatahubRestEmitter(
    gms_server="https://<your-instance>.acryl.io/gms",
    token="<your-datahub-token>"
)

# 1. Set the sibling on the correct lowercase dbt URN to point to the warehouse asset
dbt_urn = "urn:li:dataset:(urn:li:dataPlatform:dbt,my_schema.release.my_model_name,PROD)"
warehouse_urn = "urn:li:dataset:(urn:li:dataPlatform:snowflake,my_schema.release.my_model_name,PROD)"

from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.metadata.schema_classes import SiblingsClass

# Update siblings on the dbt asset
dbt_siblings = SiblingsClass(siblings=[warehouse_urn], primary=True)
mcp_dbt = MetadataChangeProposalWrapper(
    entityUrn=dbt_urn,
    aspect=dbt_siblings,
)
emitter.emit(mcp_dbt)

# 2. Update siblings on the warehouse (Snowflake) asset to point to lowercase dbt URN
warehouse_siblings = SiblingsClass(siblings=[dbt_urn], primary=False)
mcp_warehouse = MetadataChangeProposalWrapper(
    entityUrn=warehouse_urn,
    aspect=warehouse_siblings,
)
emitter.emit(mcp_warehouse)

print("Sibling relationships updated successfully.")

Step 4: Soft-Delete the Stale Uppercase URN

After the sibling relationships are corrected, soft-delete the stale uppercase dbt URN using the DataHub CLI or REST API. Soft-deleted entities are removed from search immediately and are hard-deleted after 10 days.

Using the DataHub CLI:

# Soft-delete the stale uppercase dbt URN
datahub delete \
  --urn "urn:li:dataset:(urn:li:dataPlatform:dbt,MY_SCHEMA.RELEASE.my_model_name,PROD)" \
  --soft

Or using the REST API directly:

curl -X POST \
  "https://<your-instance>.acryl.io/gms/entities?action=delete" \
  -H "Authorization: Bearer <your-datahub-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "urn": "urn:li:dataset:(urn:li:dataPlatform:dbt,MY_SCHEMA.RELEASE.my_model_name,PROD)"
  }'

Step 5: Verify Lineage Is Restored

Navigate to the warehouse asset (e.g., the Snowflake table) in the DataHub UI.
Open the Lineage tab in the default merged sibling view.
Confirm that downstream assets (exposures, downstream consumers, etc.) are now visible without needing to switch to the individual dbt component view.
Confirm that the Composed of section references the lowercase dbt URN only.

Additional Notes

Root cause: Changing convert_urns_to_lowercase from false to true causes dbt ingestion to emit assets under new lowercase URNs. Without stateful ingestion enabled, the old uppercase URNs are never cleaned up, resulting in duplicate entities. The sibling pointers remain on the old URN until manually reassigned.
Soft-delete vs. hard-delete: Soft-deleted entities are hidden from search but remain accessible via direct URN link. They are permanently hard-deleted after 10 days. If you need to hard-delete immediately, use the --hard flag with the DataHub CLI (use with caution).
Stateful ingestion timing: After enabling stateful ingestion, stale assets are only removed after two successful ingestion pipeline runs — not immediately on the first run.
Known product limitation: There is an open feature request to surface downstream lineage (such as dbt exposures) on the merged sibling view automatically. Until that is resolved, downstream lineage that is exclusively attached to the dbt entity may require navigating to the dbt sibling component view. Ensuring correct sibling pointers (as described above) is the best current mitigation.
Scope of impact: If convert_urns_to_lowercase was changed globally, many dbt models may be affected simultaneously. Audit all dbt assets for duplicate URN pairs before performing bulk remediation.
This issue applies to dbt ingestion using both the CLI and the DataHub UI-based ingestion scheduler.