Cross-Platform Lineage and URN Mapping

Lineage between different platforms (e.g., S3 to Snowflake, BigQuery to Looker) is not connecting properly due to URN mismatches or platform mapping issues. Understanding URN construction and platform mapping is essential.

You Might Be Asking:

• How do I create lineage between different platforms?
• Why isn't my cross-platform lineage connecting?
• How do I map platform identifiers?

1. Understand URN structure for cross-platform lineage:

# Basic URN format:
urn:li:dataset:(urn:li:dataPlatform:PLATFORM,DATASET_PATH,ENV)

# Examples:
# S3:
urn:li:dataset:(urn:li:dataPlatform:s3,s3://bucket/path/to/file,PROD)

# Snowflake:
urn:li:dataset:(urn:li:dataPlatform:snowflake,database.schema.table,PROD)

# BigQuery:
urn:li:dataset:(urn:li:dataPlatform:bigquery,project.dataset.table,PROD)

# Key: DATASET_PATH must match exactly between source and target

2. Map S3 to Snowflake external tables:

Example for S3 and Snowflake integration. This can be adapted for other cloud storage and data warehouse combinations like GCS to BigQuery or Azure Blob to Synapse.

# Ingest Snowflake with external table mapping
source:
  type: snowflake
  config:
    include_external_tables: true

    # This creates lineage from S3 location to Snowflake table
    # Based on external table definition

# Separately ingest S3
source:
  type: s3
  config:
    path_specs:
      - include: "s3://bucket/data/**/*.parquet"

    # Must use same naming convention
    table_pattern: "s3://bucket/data/{table}/**"

3. Create manual cross-platform lineage:

Example for S3 to Snowflake lineage. This approach can be adapted for any cross-platform lineage scenario.

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.metadata.schema_classes import UpstreamLineageClass, UpstreamClass
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter("http://localhost:8080")

# Define lineage from S3 to Snowflake
s3_urn = make_dataset_urn("s3", "my-bucket/analytics/customers")
snowflake_urn = make_dataset_urn("snowflake", "analytics_db.public.customers")

upstream_lineage = UpstreamLineageClass(
    upstreams=[
        UpstreamClass(
            dataset=s3_urn,
            type="COPY"  # or "TRANSFORMED"
        )
    ]
)

emitter.emit_mcp(
    entity_urn=snowflake_urn,
    aspect_name="upstreamLineage",
    aspect=upstream_lineage
)

4. Map BI tool connections to data platforms:

Examples for Looker and Tableau. This can be adapted for other BI tools like Power BI, Qlik, or Sisense.

# Looker to Snowflake
source:
  type: looker
  config:
    # Map Looker connection names to DataHub platforms
    connection_to_platform_map:
      "production_snowflake": "snowflake"
      "analytics_bigquery": "bigquery"

    # Map to platform instances if used
    platform_instance_map:
      "production_snowflake": "prod_snowflake"
      "analytics_bigquery": "prod_bigquery"

# Tableau to databases
source:
  type: tableau
  config:
    database_connection_platform_map:
      "Snowflake Analytics": "snowflake"
      "BigQuery Prod": "bigquery"

5. Handle platform-specific identifiers:

Example for normalizing identifiers across Snowflake, BigQuery, and PostgreSQL. This can be adapted for other databases with different casing and identifier rules.

# Helper function to normalize identifiers across platforms
def normalize_table_identifier(platform, identifier):
    """
    Normalize table identifiers to match across platforms
    """
    if platform == "snowflake":
        # Snowflake: uppercase by default
        return identifier.upper()
    elif platform == "bigquery":
        # BigQuery: case-sensitive, project.dataset.table
        return identifier.lower()
    elif platform == "postgres":
        # Postgres: lowercase unless quoted
        return identifier.lower()
    return identifier

# Usage when creating lineage
source_table = normalize_table_identifier("snowflake", "ANALYTICS.PUBLIC.CUSTOMERS")
target_table = normalize_table_identifier("bigquery", "project.dataset.customers")

6. Debug URN mismatches:

# Query to find exact URN
curl -X POST http://localhost:8080/api/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{
      search(input: {type: DATASET, query: \"customers\"}) {
        searchResults {
          entity {
            urn
            ... on Dataset {
              name
              platform { name }
            }
          }
        }
      }
    }"
  }' | jq '.data.search.searchResults[].entity.urn'

# Compare URNs from different sources
# Ensure they match exactly for lineage to connect

URN matching is case-sensitive and must be exact. Use consistent naming conventions across platforms. Platform instance helps disambiguate when multiple instances of the same platform exist. Test URN formats before large-scale lineage emission.

• URN Structure
• Cross-Platform Lineage