Lineage Missing After Adding Platform Instance to Databricks or Fivetran Ingestion – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Cross-Platform Lineage / URN Mismatch

Issue

When running both a Fivetran ingestion recipe and a Databricks ingestion recipe, lineage between Fivetran-sourced datasets and their Databricks destinations may not appear in the DataHub UI even though both ingestion runs complete successfully. This typically occurs when a platform_instance is added to one recipe but not the other, or when there is a mismatch in how dataset URNs are constructed between the two connectors. Because DataHub's lineage engine connects assets by exact URN match, any discrepancy in platform instance, catalog casing, or environment between the two recipes causes lineage edges to be silently dropped.

Error Messages

Destination URN could not be constructed
num_lineage_edges_skipped: > 0 (visible in the Fivetran ingestion report)

You Might Be Asking

Why does my Fivetran → Databricks lineage disappear after I update my ingestion recipe?
Both my Fivetran and Databricks ingestions ran successfully — why is there no lineage in the UI?
Do I need to re-run both ingestion sources after changing a platform_instance?
Why are my Databricks datasets showing up as separate entities from what Fivetran references?

Solution

The root cause is a URN mismatch between what the Fivetran connector emits for Databricks destination datasets and what the Databricks connector emits for those same datasets. There are three common variants of this mismatch, each with a specific fix:

Root Cause 1: Platform Instance Added to One Recipe but Not the Other

This is the most common trigger. When a platform_instance is added to the Databricks recipe, the URNs for all Databricks datasets change from:

urn:li:dataset:(urn:li:dataPlatform:databricks,catalog.schema.table,PROD)

to:

urn:li:dataset:(urn:li:dataPlatform:databricks,<platform_instance>.catalog.schema.table,PROD)

If the Fivetran recipe's destination_to_platform_instance mapping does not include the same platform_instance value, Fivetran will emit lineage edges pointing to the old URN format, and no lineage will be displayed.

Fix: Ensure the Fivetran recipe's destination_to_platform_instance block specifies the same platform_instance value used in the Databricks recipe. After aligning the configuration, re-run the Databricks ingestion so its datasets are re-emitted with the updated URNs. Re-running Fivetran is typically not required if its lineage edges already reference the correct (new) URN format.

# fivetran recipe (fivetran_recipe.yaml)
source:
  type: fivetran
  config:
    # ... other config ...
    destination_to_platform_instance:
      <fivetran-destination-id>:
        platform_instance: <your-platform-instance>
        env: PROD
        database: <your-catalog-name>

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.datahubproject.io/api/gms"
    token: "<your-datahub-token>"

# databricks recipe (databricks_recipe.yaml)
source:
  type: databricks
  config:
    # ... other config ...
    platform_instance: <your-platform-instance>
    env: PROD

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.datahubproject.io/api/gms"
    token: "<your-datahub-token>"

Root Cause 2: Case Sensitivity Mismatch in Catalog Name

The Fivetran connector has a database_lowercase option that defaults to true, meaning it lowercases the Databricks catalog name when constructing dataset URNs. The Databricks connector preserves the original casing returned by the Unity Catalog API. If the catalog name contains uppercase letters, this produces a silent mismatch:

# Fivetran emits (database_lowercase: true by default):
urn:li:dataset:(urn:li:dataPlatform:databricks,main.public.customers,PROD)

# Databricks emits (original casing preserved):
urn:li:dataset:(urn:li:dataPlatform:databricks,Main.public.customers,PROD)

Fix: Set database_lowercase: false in the Fivetran recipe's destination_to_platform_instance config to preserve exact catalog casing and match the URNs emitted by the Databricks connector.

# fivetran recipe — disable lowercasing to match Databricks casing
source:
  type: fivetran
  config:
    destination_to_platform_instance:
      <fivetran-destination-id>:
        platform_instance: <your-platform-instance>
        env: PROD
        database: <YourCatalogName>   # use exact case as it appears in Unity Catalog
        database_lowercase: false

Root Cause 3: Databricks Destination Catalog Not Discovered

Without Fivetran REST API credentials (api_config), the Fivetran source cannot auto-discover the Databricks destination's catalog name. If no manual override is provided via destination_to_platform_instance, lineage edges are silently skipped and the ingestion report will show num_lineage_edges_skipped > 0.

Fix: Either provide api_config with valid Fivetran API credentials so the connector can auto-discover the destination catalog, or explicitly configure destination_to_platform_instance with the correct Databricks destination ID and catalog name.

# Option A — provide Fivetran API credentials for auto-discovery
source:
  type: fivetran
  config:
    fivetran_log_config:
      # ... log config ...
    api_config:
      fivetran_api_key: "<your-fivetran-api-key>"
      fivetran_api_secret: "<your-fivetran-api-secret>"

# Option B — manually specify destination mapping
source:
  type: fivetran
  config:
    destination_to_platform_instance:
      <fivetran-destination-id>:
        platform_instance: <your-platform-instance>
        env: PROD
        database: <your-catalog-name>

Debugging Steps

Open the Fivetran ingestion report in DataHub and check the value of num_lineage_edges_skipped. If it is greater than 0, lineage edges are being dropped.
Search the Fivetran ingestion logs for Destination URN could not be constructed warnings.
Navigate to a table that should appear in lineage on both sides. Compare the exact URN shown in the browser URL for each asset — confirm that the platform, catalog casing, and platform_instance segments are identical between the Fivetran-referenced URN and the Databricks-emitted URN.
Confirm that the Databricks ingestion report lists the same schema and table names that the Fivetran connector is writing to.
After aligning the recipes, re-run the Databricks ingestion. Re-running Fivetran is only necessary if its lineage edge configuration was also changed.

Additional Notes

When a platform_instance is added to an existing Databricks recipe, all previously ingested dataset URNs will change on the next ingestion run. This is expected behavior — the old URNs (without the platform instance prefix) will become stale and lineage will not resolve until the Databricks ingestion is re-run with the updated configuration. In most cases, only the Databricks ingestion needs to be re-run; the Fivetran lineage edges already reference the new URN format if destination_to_platform_instance was configured correctly beforehand. Both recipes must use the same env value (e.g., PROD) for URNs to match. The database_lowercase: true default in the Fivetran connector is a known source of silent mismatches when Unity Catalog names contain uppercase characters.