Trino/Presto Query-Based Lineage Extraction

Lineage from Trino/Presto query logs is not being extracted correctly, or queries across catalogs are not showing proper lineage relationships. This typically occurs due to catalog mapping issues or insufficient query history access.

You Might Be Asking:

• How do I extract lineage from Trino queries?
• Why isn't cross-catalog lineage working?
• How do I map Trino catalogs to DataHub platforms?

1. Configure Trino source with query lineage:

Example for Trino. This can be adapted for Presto or other distributed SQL query engines.

source:
  type: trino
  config:
    host_port: "trino.company.com:8080"
    username: "datahub"

    # Catalog to platform mapping
    catalog_to_platform_map:
      hive: "hive"
      postgresql: "postgres"
      mysql: "mysql"
      iceberg: "iceberg"

    # Enable lineage extraction
    include_table_lineage: true
    include_view_lineage: true

    # Query history settings
    include_usage_stats: true
    query_log:
      enabled: true
      lookback_days: 7

2. For Trino with multiple catalogs:

Example for Trino with Snowflake and BigQuery catalogs. This can be adapted for other catalog configurations.

source:
  type: trino
  config:
    catalog_to_platform_map:
      # Map each Trino catalog to DataHub platform
      prod_snowflake: "snowflake"
      staging_snowflake: "snowflake"
      bigquery_analytics: "bigquery"

    # Use platform instances to differentiate
    platform_instance_map:
      prod_snowflake: "prod_snowflake"
      staging_snowflake: "staging_snowflake"

3. Extract lineage from Trino event logs:

Example for Trino. This can be adapted for Presto or other distributed SQL query engines.

source:
  type: trino
  config:
    # If using Trino event listener logs
    event_log_path: "/var/log/trino/event-logs"

    # Or query system tables
    system_tables:
      enabled: true
      query_history_table: "system.runtime.queries"

4. Handle cross-catalog queries:

Example for Trino with multiple catalogs. This can be adapted for Presto or other distributed SQL query engines.

-- Example Trino query across catalogs
INSERT INTO hive.analytics.customer_summary
SELECT 
  c.customer_id,
  SUM(o.amount)
FROM postgresql.public.customers c
JOIN mysql.sales.orders o ON c.id = o.customer_id
GROUP BY c.customer_id;

-- DataHub will create lineage:
-- postgresql.public.customers -> hive.analytics.customer_summary
-- mysql.sales.orders -> hive.analytics.customer_summary

5. Debug lineage extraction:

# Check ingestion logs for SQL parsing
datahub ingest -c trino-recipe.yml 2>&1 | grep -i "lineage\|sql"

# Verify catalog mappings are correct
# Check that source table URNs match existing datasets in DataHub

Trino lineage quality depends on SQL parser's ability to handle complex queries. Cross-catalog queries work best when all catalogs are properly mapped. Consider supplementing with view-based lineage for Trino views.

• Trino Source
• Presto Source