Duplicate Containers/Folders from dbt Ingestion Creating Shell Entities – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt / Data Warehouse Container Deduplication

Issue

When running dbt ingestion alongside a data warehouse source (such as BigQuery or Redshift), users may observe duplicate project folders or schema containers appearing in the DataHub sidebar browse tree. A second, smaller "ghost" folder appears alongside the primary, correctly populated folder. This occurs because the dbt connector, when it references a source table that has not yet been ingested by the warehouse connector, creates a shell (stub) entity for that table — including a new container hierarchy — rather than linking to the existing container. The result is two separate containers representing the same logical database, project, or schema. Deleting the duplicate entities manually and re-running ingestion often causes them to reappear if the underlying cause is not addressed.

You Might Be Asking

Why do I see two versions of the same BigQuery project or Redshift database in my DataHub sidebar?
Why do duplicate folders come back every time I re-run ingestion after deleting them?
What does dbt ingestion have to do with creating extra containers for my data warehouse?
How do I permanently remove shell entities created by dbt source ingestion?
Why do ephemeral or temporary staging tables appear as stray datasets in a separate browse folder?

Solution

There are two primary strategies to resolve this issue. Choose the approach that best fits your environment.

Option A (Recommended): Disable dbt source entity ingestion using entities_enabled.sources
Add the following configuration to your dbt ingestion recipe to prevent the dbt connector from emitting shell entities for source tables that are not yet present in DataHub:
```
source:
  type: dbt
  config:
    # ... your existing dbt config ...
    entities_enabled:
      sources: "NO"
```
After updating the recipe, manually delete the existing shell entities and their stale containers. They will not be recreated on subsequent ingestion runs.
Option B: Ingest the referenced source assets via their native warehouse connector
Run a BigQuery, Redshift, or other warehouse ingestion recipe that covers the source tables referenced by your dbt models. Once the real entities exist in DataHub with the correct container hierarchy, the dbt connector will resolve its references to them rather than creating shell entities.
```
source:
  type: bigquery   # or redshift, snowflake, etc.
  config:
    # ... your existing warehouse config ...
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
```
Stabilize platform_instance across all ingestion recipes
Container URNs are computed from a combination of database/project identifiers, platform, and platform_instance. If different recipes or tools (e.g., an Airflow plugin) use inconsistent platform_instance values — or if the value changes between runs — DataHub will generate distinct container URNs for the same logical entity, producing additional duplicate folders.

Ensure all recipes and pipeline integrations that reference the same data platform use an identical, stable platform_instance value:
```
source:
  type: redshift   # or bigquery, etc.
  config:
    platform_instance: "my-warehouse-instance"  # use the same value everywhere
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
```
If you use the DataHub Airflow plugin to emit lineage, verify that the platform_instance argument passed in your DAG inlets/outlets matches the value configured in your warehouse ingestion recipe:
```
# Example Airflow lineage declaration — platform_instance must match the ingestion recipe
inlets=[Dataset(platform="redshift", name="<db>.<schema>.<table>", platform_instance="<your-platform-instance>")],
outlets=[Dataset(platform="s3", name="<your-s3-path>")]
```
Enable stateful ingestion to prevent stale entity accumulation
Without stateful ingestion, deleted or orphaned entities can reappear on every run. Enable stateful ingestion and stale metadata removal in all relevant recipes. Also ensure ignore_old_state is not set to true, as that disables checkpoint tracking and defeats stale removal:
```
source:
  type: redshift   # or dbt, bigquery, etc.
  config:
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
      # Do NOT set ignore_old_state: true — this disables checkpoint tracking
pipeline_name: "<your-unique-pipeline-name>"  # required for stateful ingestion
```
Hard-delete stale shell entities using the DataHub CLI
Soft deletes (via the UI "Remove All" action) hide entities but do not purge their URNs. If the same URN is re-emitted by a subsequent ingestion run, the entity reappears. Use the CLI to perform a hard delete of orphaned URNs. Always run with --dry-run first to confirm the target set before committing:
```
# Preview what will be deleted — always run this first
datahub delete --urn "urn:li:dataset:(urn:li:dataPlatform:<platform>,<stale-urn-value>,PROD)" --hard --dry-run

# Hard delete a specific stale entity
datahub delete --urn "urn:li:dataset:(urn:li:dataPlatform:<platform>,<stale-urn-value>,PROD)" --hard

# Hard delete a list of URNs from a file (one URN per line)
datahub delete --urn-file stale_urns.txt --hard
```
Note: The --urn-like flag does not exist in the DataHub CLI. Use --urn for a single URN or --urn-file for a batch.
Suppress ephemeral or date-partitioned staging tables using table_pattern.deny
If your data pipelines create temporary or date-suffixed staging tables (e.g., stg_mytable_20240101) that are dropped after each run, those tables can accumulate as stray orphaned entities in DataHub. Prevent them from being ingested in the first place by adding deny patterns to your warehouse ingestion recipe:
```
source:
  type: redshift   # or bigquery, etc.
  config:
    table_pattern:
      deny:
        - ".*stg_<table_prefix>_[0-9]{4}-[0-9]{2}-[0-9]{2}.*"   # YYYY-MM-DD suffix
        - ".*stg_<table_prefix>_[0-9]{8}.*"                       # YYYYMMDD suffix
```
Address duplicate containers produced by the dbt target_platform sibling behavior
When a dbt ingestion recipe is configured with target_platform: <warehouse>, the dbt connector emits both a canonical dbt-platform entity and a "sibling" entity on the target warehouse platform for each model. The sibling entity does not include container metadata, so if the corresponding physical table does not exist in the warehouse schema currently being ingested, the sibling will appear as an orphan in a stray browse folder. To resolve this, ensure the dbt models in question are materialized as tables or views in the warehouse (not ephemeral), and re-run the warehouse ingestion so it can write the correct container aspect onto those URNs.

Additional Notes

A known bug in certain DataHub versions caused duplicate folders to be created more aggressively during dbt ingestion; this was addressed in DataHub v1.1 and later releases — upgrading is recommended if you are on an older version. The ignore_old_state: true setting under stateful_ingestion completely disables checkpoint-based stale removal and should be removed from production recipes. For BigQuery specifically, avoid setting include_data_platform_instance: true in your BigQuery ingestion recipe; this setting is documented to cause browse path redundancy because the BigQuery project ID already functions as the top-level container. If you use the Airflow DataHub plugin, be aware that auto-extractors (enabled via AIRFLOW__DATAHUB__ENABLE_EXTRACTORS=True) will automatically emit lineage for operators even when no explicit inlets/outlets are declared; misconfigured or inconsistent platform_instance values in DAG declarations are a common secondary source of duplicate stray containers. Setting AIRFLOW__DATAHUB__ENABLE_EXTRACTORS=False disables auto-extraction globally across all DAGs — evaluate the impact before applying this change.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation