dbt Ingestion Missing Metadata Due to Stale Artifacts in S3 – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt Source — Artifact Staleness and Metadata Mapping

Issue

When using the DataHub dbt connector with artifacts stored in cloud object storage (e.g., S3), metadata defined in models.yml — such as owners, domains, and column-level properties — may not appear in DataHub even after multiple ingestion runs. This happens because DataHub reads from the compiled manifest.json and catalog.json artifact files, not directly from the dbt repository. If those artifact files are stale (i.e., they were not regenerated after recent changes to models.yml), the new metadata is simply absent from what DataHub processes. Re-running the ingestion recipe against the same stale artifacts will not resolve the issue. Additionally, when incremental ownership is disabled (the default), ownership aspects are fully overwritten on every run, meaning models whose owner fields are absent from a stale manifest may have their existing owners cleared in DataHub.

Error Messages

Node missing from catalog: <model_name> — catalog was not generated by dbt docs generate and is incomplete

You Might Be Asking

Why is metadata I added to models.yml not showing up in DataHub even after running ingestion multiple times?
Why do some dbt models have owners or domains in DataHub but others do not, even though all are defined in the YAML?
Why did owners disappear from a DataHub asset after a dbt ingestion run?
Why does the ingestion report show zero domains aspects emitted across all datasets?

Solution

Regenerate dbt artifacts before every ingestion run.
DataHub reads metadata exclusively from the compiled manifest.json and catalog.json files. Run both of the following dbt commands and ensure the output files are uploaded to your artifact storage location (e.g., S3) before the DataHub ingestion job fires:
```
dbt compile
dbt docs generate
```
If your artifact upload is a separate pipeline step, verify it cannot run out of order with repository merges. The safest pattern is a single sequential job: compile → docs generate → upload artifacts → trigger DataHub ingestion.

Verify domain metadata uses the correct YAML path.

The DataHub dbt connector extracts domain metadata only when it is nested under the datahub key inside meta. Using meta.domain directly is silently ignored.

Correct (will be picked up):

models:
  - name: my_model
    meta:
      datahub:
        domain: "Engineering"
        # or use a fully qualified URN:
        # domain: "urn:li:domain:Engineering"

Incorrect (silently ignored):

models:
  - name: my_model
    meta:
      domain: "Engineering"   # NOT picked up by the DataHub connector

Verify owner metadata uses a supported YAML path.

The connector supports two formats. If both are present, meta.datahub.owners takes precedence and meta.owner is ignored for that model.

Option 1 — Simple string (no namespace required):

models:
  - name: my_model
    meta:
      owner: "user@example.com"

Option 2 — DataHub-namespaced (supports ownership types and multiple owners):

models:
  - name: my_model
    meta:
      datahub:
        owners:
          - owner: "urn:li:corpuser:user@example.com"
            type: "TECHNICAL_OWNER"
          - owner: "urn:li:corpuser:other@example.com"
            type: "DATA_STEWARD"

Enable incremental ownership to prevent owners from being cleared.
By default, ownership is emitted as a full UPSERT on every run. If a model's owner fields are absent from the manifest (e.g., because the manifest is stale or the fields are simply not defined), DataHub will overwrite that asset's ownership with an empty list, removing any previously set owners. Setting incremental_ownership: true switches to PATCH semantics — owners are only added, never removed by the ingestion run.

Add the following to your dbt ingestion recipe:
```
source:
  type: dbt
  config:
    manifest_path: "s3://<your-bucket>/dbt/manifest.json"
    catalog_path: "s3://<your-bucket>/dbt/catalog.json"
    incremental_ownership: true
    # ... other config options
```
Account for domain transformer lag (one additional run).
If you use a two-step approach where a meta_mapping rule first converts a meta field into a tag, and a second transformer (e.g., domain_mapping_based_on_tags) converts that tag into a domain assignment, be aware of the following behavior: the second transformer reads tags from DataHub's already-persisted metadata, not from the tags produced earlier in the same ingestion run. This means a newly added domain will be reflected as a tag after the first run and fully applied as a domain after the second run. This is expected behavior and is resolved by running ingestion twice after the artifact files have been updated.
Confirm the fix by reviewing the ingestion report.
After updating your artifact generation pipeline and re-running ingestion, open the DataHub ingestion report and verify that domains and ownership aspect counts have increased for the affected datasets. The presence of Node missing from catalog warnings in earlier runs is a reliable indicator that the catalog artifact was incomplete at the time of ingestion.

Additional Notes

The root cause described here — stale artifacts in object storage — is non-obvious because the ingestion pipeline itself completes successfully with no errors, and the missing metadata is silently absent rather than logged as a failure. The ingestion report's aspect counts (e.g., zero domains aspects emitted across all datasets) are the clearest diagnostic signal. Always inspect the ingestion report alongside the ingestion logs when troubleshooting missing dbt metadata. Note also that dbt compile alone does not produce the catalog; dbt docs generate is required to produce a complete catalog.json. Running only dbt compile will result in Node missing from catalog warnings and incomplete column-level metadata in DataHub.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation