dbt Ingestion Issues and Model Linking

dbt model ingestion fails or dbt models are not properly linked to their materialized tables in the data warehouse. This prevents proper lineage tracking and metadata association between dbt models and physical database objects.

Common Error Messages:

• 401 Unauthorized error

Note: For DataHub Cloud customers, infrastructure-level changes (Kubernetes, AWS resources, operating system configurations) are managed by DataHub Support. The solutions below focus on configuration changes that customers can implement.

1. Ensure dbt manifest and catalog files are accessible:

   • Verify manifest.json and catalog.json paths are correct in the ingestion recipe
   • Confirm the files were generated by a recent dbt run
   • Check file permissions allow the ingestion executor to read them

2. Configure dbt source with proper linking:

source:
  type: dbt
  config:
    manifest_path: "target/manifest.json"
    catalog_path: "target/catalog.json"
    target_platform: "snowflake"  # or bigquery, postgres, etc.
    target_platform_instance: "prod"
    load_schemas: true
    node_type_pattern:
      allow: ["model", "source", "test"]

3. Link dbt models to warehouse tables:

   • Use GraphQL API to create lineage between dbt models and materialized tables
   • Ensure the target_platform matches your actual data warehouse platform
   • Verify URNs match between dbt models and physical tables

4. Troubleshoot model linking issues:

   • Check that database/schema names in dbt match the warehouse naming
   • Verify target_platform_instance matches the instance name used in warehouse ingestion
   • Use the env configuration to match environments across ingestion sources

Always backup your DataHub configuration before making changes. Test solutions in a non-production environment when possible.

• Ingestion Sources
• dbt Integration

• 5720, 5921, 5790, 5716, 5715, 5674, 5631, 5566, 5047, 5242, 5584