Column-Level Lineage Not Extracting

Column-level lineage (showing which source columns feed into which target columns) is not being extracted or displayed, even though table-level lineage works. This commonly occurs because column lineage extraction is expensive and requires specific configuration or SQL parsing capabilities.

You Might Be Asking:

• How do I enable column-level lineage?
• Why is column lineage missing when table lineage works?
• Which connectors support column-level lineage?

1. Enable column lineage in source configuration:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    # Column lineage requires additional configuration
    include_column_lineage: true

    # Requires query history analysis
    include_usage_stats: true
    include_operational_stats: true

    # May need longer lookback for sufficient query samples
    start_time: "-30d"

2. Verify connector supports column lineage:

Supported:
- Snowflake (via query history)
- BigQuery (via query history)
- Redshift (limited support)
- dbt (from manifest)

Limited/Not Supported:
- Most other databases (table lineage only)
- File-based sources

3. For dbt, ensure load_schemas is enabled:

Example for dbt. This can be adapted for other transformation tools.

source:
  type: dbt
  config:
    manifest_path: "target/manifest.json"
    catalog_path: "target/catalog.json"

    # Required for column lineage
    load_schemas: true

    # Parse column lineage from SQL
    parse_table_names_from_sql: true

4. Manual column lineage via API:

Example for Snowflake. This can be adapted for other data sources/connectors.

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.metadata.schema_classes import (
    FineGrainedLineageClass,
    FineGrainedLineageUpstreamClass,
    FineGrainedLineageDownstreamClass
)

column_lineage = FineGrainedLineageClass(
    upstreamType="FIELD_SET",
    upstreams=[
        FineGrainedLineageUpstreamClass(
            dataset="urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.source,PROD)",
            fields=["urn:li:schemaField:(urn:li:dataset:...,source_col)"]
        )
    ],
    downstreams=[
        FineGrainedLineageDownstreamClass(
            dataset="urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.target,PROD)",
            fields=["urn:li:schemaField:(urn:li:dataset:...,target_col)"]
        )
    ]
)

5. Use SQL parser for view/transformation lineage:

source:
  config:
    # Enable SQL parsing for views
    parse_view_ddl: true
    include_view_lineage: true

    # Choose SQL parser
    sql_parser: "sqlglot"  # More comprehensive than sql-metadata

Column lineage extraction can significantly increase ingestion time and resource usage. For Snowflake/BigQuery, sufficient query history is needed to build accurate column lineage. Consider the trade-off between completeness and performance.

• Column-level Lineage
• Snowflake Column Lineage