dbt Core platform_instance Best Practices in Multi-Account Environments – DataHub Customer Support Portal

Area: Best Practices

Sub-Area: dbt Core Ingestion Configuration

Issue

When running dbt Core ingestion across multiple accounts, environments, or warehouse connections, choosing an appropriate platform_instance value is critical for ensuring stable, unique asset identifiers in DataHub. Without a deliberate strategy, teams may encounter duplicate URNs, missing lineage, or broken asset references when the same dbt project is ingested from multiple contexts or when ingestion configurations change over time.

You Might Be Asking

What should I use as the platform_instance for my dbt Core ingestion recipe?
How do I avoid URN collisions when ingesting dbt from multiple environments (dev, staging, prod)?
Should I use a hash of my connection details as the platform_instance?
What happens if my platform_instance value changes between ingestion runs?
How do I keep dbt lineage intact across multiple warehouse accounts?

Solution

Understand the role of platform_instance.
In DataHub, platform_instance is used to differentiate assets that share the same platform (e.g., Snowflake, BigQuery) but belong to different logical deployments. For dbt Core, it also scopes the URNs generated for dbt models, sources, and seeds. A stable, unique value is essential because changing it will cause DataHub to treat assets as new entities, breaking lineage and creating orphaned nodes.

Use a human-readable, environment-scoped identifier.

The recommended approach is to compose the platform_instance from stable, meaningful components such as the environment name, warehouse account identifier, or logical deployment name. Avoid values derived from transient or frequently changing properties.

Recommended pattern:

# datahub-dbt-recipe.yml

source:
  type: dbt
  config:
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    target_platform: snowflake

    # Use a stable, human-readable identifier scoped to the environment
    platform_instance: "<warehouse-account-id>-prod"

    # Ensure the warehouse platform_instance matches
    platform_instance_map:
      snowflake: "<warehouse-account-id>-prod"

Example values by environment:

# Development environment
platform_instance: "my-org-snowflake-dev"

# Staging environment
platform_instance: "my-org-snowflake-staging"

# Production environment
platform_instance: "my-org-snowflake-prod"

Align platform_instance with your warehouse ingestion recipe.

For dbt lineage to resolve correctly, the platform_instance used in the dbt recipe must match the platform_instance used when ingesting the underlying warehouse (e.g., Snowflake, BigQuery, Redshift). Mismatches between these two values are the most common cause of broken upstream/downstream lineage in DataHub.

# Snowflake ingestion recipe (snowflake-recipe.yml)
source:
  type: snowflake
  config:
    account_id: "<your-snowflake-account>"
    platform_instance: "my-org-snowflake-prod"   # <-- must match dbt recipe

# dbt Core ingestion recipe (dbt-recipe.yml)
source:
  type: dbt
  config:
    target_platform: snowflake
    platform_instance: "my-org-snowflake-prod"   # <-- must match Snowflake recipe
    platform_instance_map:
      snowflake: "my-org-snowflake-prod"          # <-- must match Snowflake recipe

Multi-account strategy: one recipe per account/environment.

When dbt projects are deployed against multiple warehouse accounts (e.g., separate Snowflake accounts for prod and non-prod, or multiple BigQuery projects), create a separate ingestion recipe for each account and assign a distinct platform_instance to each.

# Recipe for Account A (production)
source:
  type: dbt
  config:
    manifest_path: /path/to/prod/manifest.json
    catalog_path: /path/to/prod/catalog.json
    target_platform: snowflake
    platform_instance: "snowflake-account-a-prod"
    platform_instance_map:
      snowflake: "snowflake-account-a-prod"

---

# Recipe for Account B (development)
source:
  type: dbt
  config:
    manifest_path: /path/to/dev/manifest.json
    catalog_path: /path/to/dev/catalog.json
    target_platform: snowflake
    platform_instance: "snowflake-account-b-dev"
    platform_instance_map:
      snowflake: "snowflake-account-b-dev"

Understand the tradeoffs of hashing-based platform_instance values.
Some teams consider auto-generating the platform_instance by hashing connection parameters (e.g., account ID, database, schema) to ensure uniqueness. While this approach can prevent collisions, it introduces significant risks:
- Instability: Any change to the hashed inputs (account migration, credential rotation affecting connection strings, recipe refactoring) will produce a different hash, causing DataHub to generate new URNs and orphan all previously ingested assets.
- Opacity: Hash-based values are not human-readable, making it difficult to identify which DataHub assets belong to which environment when browsing the catalog.
- Debugging difficulty: Lineage breaks caused by hash mismatches are hard to diagnose without access to the original recipe configuration.
For these reasons, explicitly defined, human-readable identifiers are strongly preferred over auto-generated hashes.
Never change platform_instance for an existing ingestion without a migration plan.
Changing platform_instance on an already-running ingestion pipeline will cause DataHub to create entirely new URNs for all dbt assets. Old assets will remain in the catalog as orphans unless explicitly soft-deleted. If a change is unavoidable, plan to:
- Soft-delete or remove old assets using the DataHub CLI or API before switching.
- Update all downstream ingestion recipes that reference the old platform_instance simultaneously.
- Re-run all affected ingestion sources after the change to rebuild lineage.
```
# Using the DataHub CLI to soft-delete assets under the old platform_instance
# before migrating to a new value
datahub delete \
  --platform dbt \
  --platform-instance "<old-platform-instance>" \
  --soft
```

Additional Notes

The platform_instance_map field in the dbt recipe controls how dbt source nodes are resolved to warehouse URNs. It must be kept in sync with the platform_instance used in the corresponding warehouse ingestion recipe. This configuration is available in DataHub's dbt ingestion source for both dbt Core and dbt Cloud connectors. If you are using dbt Cloud, the platform_instance can often be derived from the dbt Cloud account and project IDs for additional stability. These recommendations apply to DataHub version 0.10.x and later; earlier versions may have different URN construction behavior for dbt assets.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation