Lineage Not Appearing for Tables or Datasets

After running ingestion, lineage relationships between tables, views, or datasets are not visible in the DataHub UI. This can occur due to missing lineage extraction configuration, insufficient permissions to query metadata tables, or incompatible source versions. Even more commonly, this can be due to a Urn case mismatch in upstreamLineage aspect.

You Might Be Asking:

• Why isn't lineage showing up after ingestion?
• How do I enable lineage extraction for my data source?
• Does DataHub support lineage for my specific data platform?

1. Ensure lineage extraction is enabled in your recipe:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    include_table_lineage: true
    include_view_lineage: true
    include_column_lineage: true
    # For query-based lineage
    include_usage_stats: true
    include_operational_stats: true

2. Verify the source connector supports lineage:

   • Check the connector documentation for lineage capabilities
   • Some connectors require additional permissions or configuration

3. For SQL-based lineage, ensure query history access:

Example for Snowflake. This can be adapted for other data sources/connectors.

-- Snowflake: Grant access to query history
GRANT IMPORTED PRIVILEGES ON DATABASE snowflake TO ROLE datahub_role;

4. If using file-based lineage ingestion, verify the format:

{
  "upstream": "urn:li:dataset:(urn:li:dataPlatform:hive,database.source_table,PROD)",
  "downstream": "urn:li:dataset:(urn:li:dataPlatform:hive,database.target_table,PROD)",
  "type": "TRANSFORMED"
}

5. Check if lineage was actually ingested using the API:

curl -X POST 'http://localhost:8080/api/graphql' \
  -H 'Content-Type: application/json' \
  -d '{"query": "{ dataset(urn: \"YOUR_URN\") { upstreamLineage { entities { urn } } } }"}'

Lineage extraction can significantly increase ingestion time and resource usage. For large deployments, consider running lineage ingestion separately or on a different schedule than metadata ingestion.

• Lineage Overview
• Snowflake Lineage