Platform Instance Configuration and URN Conflicts

Assets from different environments (dev, staging, prod) or different instances of the same platform are conflicting or overwriting each other in DataHub. This occurs when platform_instance is not properly configured, causing URN collisions between separate instances.

You Might Be Asking:

• How do I separate dev and prod environments in DataHub?
• What is platform_instance and when should I use it?
• Why are my staging tables overwriting production tables?

1. Configure platform_instance for each environment:

Example for Snowflake. This can be adapted for other data sources/connectors.

# Production Snowflake
source:
  type: snowflake
  config:
    host_port: "prod.snowflakecomputing.com"
    platform_instance: "prod_snowflake"
    env: "PROD"

---
# Staging Snowflake  
source:
  type: snowflake
  config:
    host_port: "staging.snowflakecomputing.com"
    platform_instance: "staging_snowflake"
    env: "STAGING"

2. Understand resulting URN format:

Without platform_instance:
urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.table,PROD)

With platform_instance:
urn:li:dataset:(urn:li:dataPlatform:snowflake,prod_snowflake.db.schema.table,PROD)

3. Use platform_instance for multi-tenant scenarios:

Example for BigQuery. This can be adapted for other data sources/connectors.

# Customer A's BigQuery
source:
  type: bigquery
  config:
    project_id: "customer-a-project"
    platform_instance: "customer_a"

# Customer B's BigQuery
source:
  type: bigquery
  config:
    project_id: "customer-b-project"
    platform_instance: "customer_b"

4. Ensure consistency across related ingestions:

Example for Snowflake with dbt and Looker. This can be adapted for other data sources/connectors.

# Snowflake ingestion
source:
  type: snowflake
  config:
    platform_instance: "prod_snowflake"

# dbt ingestion (must match!)
source:
  type: dbt
  config:
    target_platform: "snowflake"
    target_platform_instance: "prod_snowflake"  # Must match

# Looker ingestion
source:
  type: looker
  config:
    connection_to_platform_map:
      "Production Snowflake": "snowflake"
    platform_instance_map:
      "Production Snowflake": "prod_snowflake"  # Must match

5. Migrate existing assets to use platform_instance:

# This requires custom scripting or API calls
# Contact support for migration assistance

# Example GraphQL approach (conceptual)
# 1. List all assets without platform_instance
# 2. Create new URNs with platform_instance
# 3. Migrate metadata to new URNs
# 4. Deprecate old URNs

Once assets are created without platforminstance, adding it later creates new assets rather than updating existing ones. Plan your platforminstance strategy before initial ingestion to avoid migration complexity.

• Platform Instance
• URN Structure