dbt Metadata Not Appearing in DataHub

dbt model metadata, including descriptions, tags, and lineage, is not appearing in DataHub after running the dbt ingestion source. This commonly happens due to incorrect manifest path, missing catalog file, or mismatched URN formats between dbt and other sources.

You Might Be Asking:

• How do I ingest dbt metadata into DataHub?
• Why isn't my dbt lineage showing up?
• How do I match dbt models with my data warehouse tables?

1. Configure dbt source with correct paths:

Example for dbt with Snowflake target. This can be adapted for other data warehouse targets.

source:
  type: dbt
  config:
    # Required: manifest file with model definitions
    manifest_path: "/path/to/target/manifest.json"

    # Required: catalog file with table metadata
    catalog_path: "/path/to/target/catalog.json"

    # Optional: run_results for execution info
    run_results_path: "/path/to/target/run_results.json"

    # Target platform (where dbt models are materialized)
    target_platform: "snowflake"

    # Environment
    env: "PROD"

2. Ensure manifest and catalog files are up to date:

# Generate fresh dbt artifacts
cd your-dbt-project
dbt compile  # Generates manifest.json
dbt docs generate  # Generates catalog.json

3. Match platform instance with your warehouse ingestion:

Example for dbt with Snowflake target. This can be adapted for other data warehouse targets.

source:
  type: dbt
  config:
    target_platform: "snowflake"
    platform_instance: "prod_snowflake"  # Must match Snowflake recipe

    # Configure target environment
    target_platform_instance: "prod_snowflake"

4. For dbt Cloud, use the API-based ingestion:

Example for dbt Cloud. This can be adapted for self-hosted dbt Core.

source:
  type: dbt-cloud
  config:
    account_id: "12345"
    project_id: "67890"
    job_id: "11111"
    token: "${DBT_CLOUD_TOKEN}"

5. Verify URN matching:

# Check the URN format in manifest.json
cat target/manifest.json | jq '.nodes[] | select(.name=="your_model") | .database, .schema, .name'

# Should match the pattern:
# urn:li:dataset:(urn:li:dataPlatform:snowflake,DATABASE.SCHEMA.MODEL_NAME,PROD)

dbt metadata enhances existing datasets with model documentation, ownership, and tags. Run dbt ingestion after warehouse ingestion to ensure datasets exist. Use the load_schemas option to include column-level lineage.

• dbt Integration Guide
• dbt Cloud Source