Skip to main content

Fixing Missing Lineage Caused by Case Sensitivity and Incorrect Connector Settings

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Lineage Resolution — Snowflake, dbt, Looker/LookML

Issue

Users may encounter missing upstream or downstream lineage in DataHub when ingesting assets across Snowflake, dbt, and Looker/LookML. Common symptoms include: no upstream lineage appearing for LookML views (especially those backed by derived tables), no upstream lineage for pure Snowflake models (not managed by dbt), and broken lineage chains caused by inconsistent casing of database, schema, or table names across ingestion sources. Because DataHub resolves lineage by matching entity URNs exactly, any mismatch in casing or misconfigured connector settings will silently break lineage edges between assets.

Error Messages

  • This entity is not discoverable via search or lineage graph. Contact your DataHub admin for more information.
  • Permission denied / insufficient privileges on Looker service account

You Might Be Asking

  • Why does my Snowflake table show no upstream lineage even though it exists in DataHub?
  • Why does my LookML view not have upstream lineage when the source table is clearly defined?
  • Why does lineage break between dbt and Snowflake assets when the table names look the same?
  • Does casing in database or schema names affect lineage resolution in DataHub?
  • What Looker API settings are needed for derived table lineage in LookML?

Solution

There are three independent root causes that can each independently break lineage. Address all three for a complete resolution.

  1. Fix case sensitivity mismatches across all ingestion sources (Snowflake, dbt, Looker/LookML)

    DataHub resolves lineage by matching entity URNs exactly. If one connector emits DWH_PROD.SCHEMA.TABLE while another emits dwh_prod.schema.table, DataHub treats these as completely different entities and no lineage edge is created. Ensure that convert_urns_to_lowercase: true is set consistently in all Snowflake, dbt, and Looker/LookML recipes.

    Example Snowflake recipe snippet:

    source:
  type: snowflake
  config:
    account_id: "<your-snowflake-account>"
    username: "<your-username>"
    password: "<your-password>"
    convert_urns_to_lowercase: true
    include_table_lineage: true

    Example dbt recipe snippet:

    source:
  type: dbt
  config:
    manifest_path: "<path-to-manifest.json>"
    catalog_path: "<path-to-catalog.json>"
    convert_urns_to_lowercase: true

    After updating these settings, re-run your ingestion pipelines and verify that lineage edges now appear.

  2. Enable Snowflake table-level lineage via access_history

    For pure Snowflake models (not managed by dbt), table-to-table lineage is sourced from snowflake.account_usage.access_history. This requires:

    • Snowflake Enterprise Edition or higher — Standard Edition does not populate access_history.
    • The DataHub ingestion role must have IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE.
    • include_table_lineage: true must be set in the Snowflake recipe.

    Grant the required privilege in Snowflake:

    GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE <datahub_ingestion_role>;

    Confirm your Snowflake recipe includes:

    source:
  type: snowflake
  config:
    include_table_lineage: true
    convert_urns_to_lowercase: true
  3. Enable API-based lineage resolution for LookML derived tables

    For LookML views backed by derived tables, regex-based SQL parsing is insufficient to resolve upstream lineage. You must enable the Looker Query API so DataHub retrieves fully-rendered SQL. Add the following to your LookML ingestion recipe:

    source:
  type: lookml
  config:
    use_api_for_view_lineage: true
    use_api_cache_for_view_lineage: true
    connection_to_platform_map:
      "<looker-connection-name>":
        platform: snowflake
        database: "<your-database-name>"

    Note: connection_to_platform_map is required so DataHub can resolve which Snowflake entities a LookML view references. Without it, cross-source lineage from LookML to Snowflake cannot be established.

    Warning: Enabling use_api_for_view_lineage: true significantly increases ingestion runtime. If you run multiple ingestion sources concurrently, stagger their schedules by at least 15 minutes to avoid hitting Looker API rate limits or causing memory pressure on your DataHub services.

  4. Verify Looker service account permissions

    If LookML ingestion is failing with permission-related errors, confirm that the service account used for Looker API ingestion has the following Looker permissions assigned:

    • access_data
    • see_lookml_dashboards
    • see_looks
    • see_user_dashboards
    • explore
    • see_sql
    • see_lookml
    • clear_cache_refresh
    • manage_models
    • see_datagroups
    • see_pdts
    • see_queries
    • see_schedules
    • see_system_activity
    • see_users

    Additionally, ensure the service account's role includes model-set access for all LookML projects whose lineage you want to ingest. Verify role assignments in Looker Admin → Users → API Credentials.

  5. Review the skip_sources_in_lineage setting

    The skip_sources_in_lineage parameter in dbt recipes controls whether source nodes are excluded from lineage graphs. Unless you have a specific reason to exclude source nodes, set this to false (which is the default) to ensure full lineage chains are captured:

    source:
  type: dbt
  config:
    skip_sources_in_lineage: false
    convert_urns_to_lowercase: true

Additional Notes

  • The most common and impactful root cause of silently broken lineage is case sensitivity mismatches. Always ensure convert_urns_to_lowercase: true is applied consistently across all connector types in your environment before investigating other causes.
  • Snowflake table-level lineage via access_history is only available on Enterprise Edition or higher. Standard Edition users will not see table-to-table lineage for pure Snowflake models.
  • Enabling use_api_for_view_lineage: true in LookML ingestion can significantly increase ingestion time. Plan your ingestion schedules accordingly and stagger concurrent ingestion sources.
  • The connection_to_platform_map configuration is essential for resolving cross-platform lineage between Looker and Snowflake (or other data platforms). Without it, LookML view lineage to upstream warehouse tables will not be stitched.
  • After making recipe changes, always re-run the full ingestion pipeline and allow time for the lineage graph to update before verifying results.

Related Documentation

Tags: lineage, missing-lineage, snowflake, dbt, looker, lookml, case-sensitivity, convert-urns-to-lowercase, derived-tables, service-account-permissions