Snowflake Lineage Ingestion for Snowpipes – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Snowflake Lineage

Issue

DataHub is not showing lineage for Snowflake Snowpipe operations despite having the correct permissions and data being present in the copy_history table. The lineage from S3 to Snowflake tables appears as "phantom lineage" pointing to non-existent S3 entities in DataHub.

You Might Be Asking

Why don't I see Snowpipes as first-class objects in DataHub lineage?
Why is S3-to-Snowflake lineage missing even though I have granted IMPORTED PRIVILEGES?
How do I configure my Snowflake recipe to capture Snowpipe lineage?

Solution

DataHub handles Snowpipe lineage through two separate mechanisms that need to be properly configured:

Configure S3-to-Table lineage (via include_table_lineage)

Verify these settings in your Snowflake ingestion recipe:

source:
  type: snowflake
  config:
    include_table_lineage: true  # Must be true (default)
    lookback_days: 30           # Increase if needed to cover pipe activity period
    schema_pattern:
      allow:
        - "PROD_RAW.*"          # Ensure downstream tables are included
    table_pattern:
      allow:
        - ".*"                  # Ensure all relevant tables are captured

Enable Snowpipe objects as DataJob entities

Add these configurations to capture Snowpipes as first-class objects:

source:
  type: snowflake
  config:
    include_pipes: true         # Enable Snowpipe ingestion
    include_stages: true        # Required for stage URN resolution

Grant additional Snowflake permissions

Execute these grants for the DataHub service account:

GRANT USAGE ON DATABASE  TO ROLE ;
GRANT USAGE ON SCHEMA .INFORMATION_SCHEMA TO ROLE ;
GRANT SELECT ON ALL TABLES IN SCHEMA .INFORMATION_SCHEMA TO ROLE ;
GRANT SELECT ON FUTURE TABLES IN SCHEMA .INFORMATION_SCHEMA TO ROLE ;

Set up S3 ingestion (optional but recommended)

To resolve "phantom lineage" and show complete end-to-end lineage, configure an S3 source in DataHub:

source:
  type: s3
  config:
    bucket_name: 
    aws_config:
      aws_region: 
    path_specs:
      - include: "s3:///path/to/snowpipe/data/**"

Verify time window coverage

Check that your lookback window covers the period when Snowpipes were active by querying:

SELECT DISTINCT pipe_name, stage_location, 
       concat(table_catalog_name, '.', table_schema_name, '.', table_name) as downstream_table,
       last_load_time
FROM snowflake.account_usage.copy_history 
WHERE last_load_time > CURRENT_DATE - 30
AND status IN ('Loaded', 'Partially loaded') 
ORDER BY pipe_name;

Additional Notes

The include_pipes setting defaults to false to keep DataJob graphs clean for users who don't need pipe-level lineage. S3 ingestion is optional but recommended for complete lineage visibility. If you see phantom lineage edges, it indicates that DataHub detected the relationship from copy_history but the upstream S3 entities haven't been ingested. Time window mismatches are a common cause of missing lineage - ensure lookback_days covers your pipe activity period.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation