Removing a Platform Instance from DataHub: Hard Delete and Stateful Ingestion Cleanup – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Platform Instance Removal and Orphaned Entity Cleanup

Issue

When attempting to remove a platform instance from DataHub (for example, after renaming or consolidating Redshift ingestion configurations), users find that the platform instance continues to appear in the UI filter dropdowns even after hard deleting the platform instance entity via the API. This occurs because the UI dropdown is populated from aggregations across all datasets that reference the platform instance via their dataPlatformInstance aspect — not from the platform instance entity itself. Simply deleting the platform instance container entity does not cascade to remove references held by the datasets, containers, and other entities associated with that instance. Additionally, if a hard delete is performed on the platform instance entity before recursively removing its children, those child entities become orphaned and cannot be cleaned up by standard stateful ingestion or recursive delete commands without additional intervention.

You Might Be Asking

Why does a platform instance still appear in the UI after I hard deleted it via the API?
How do I fully remove a platform instance and all its associated entities from DataHub?
Why did datahub delete --recursive not remove the platform instance after I already hard deleted it?
What is the correct order of operations for removing a platform instance?
How does stateful ingestion help prevent orphaned entities when removing a platform instance?

Root Cause

The UI platform instance filter dropdown is driven by an aggregateAcrossEntities query that runs an Elasticsearch aggregation over the dataPlatformInstance.instance field across all indexed entities. As long as any dataset, container, or other entity still holds a dataPlatformInstance aspect referencing the deleted instance, the instance name will continue to appear in the dropdown. Deleting the platform instance entity itself only removes that single entity — it does not cascade to clean up the referencing entities. Furthermore, if the platform instance entity is hard deleted first, subsequent recursive delete commands lose their anchor and may fail to enumerate the orphaned child entities, leaving them stranded in the index.

Solution

Preferred approach — Enable stateful ingestion before removing the platform instance. Update your ingestion recipe to enable stateful ingestion with stale metadata removal before removing the platform_instance configuration. When you then run ingestion without the platform instance, DataHub will automatically remove entities that are no longer produced.
```
source:
  type: redshift
  config:
    # ... other configs ...
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
```
After saving the recipe, remove the platform_instance field and re-run ingestion. DataHub will mark the old platform-instance-scoped entities as stale and delete them cleanly.
Alternative approach — Recursive hard delete via the CLI. If stateful ingestion is not suitable for your use case, use the DataHub CLI to recursively hard delete the platform instance and all entities that reference it. Run this command before performing any manual hard deletes on the platform instance entity itself:
```
datahub delete \
  --urn 'urn:li:dataPlatformInstance:(urn:li:dataPlatform:<your-platform>,<your-instance-name>)' \
  --recursive \
  --hard \
  --force
```
Replace <your-platform> (e.g., redshift, mysql) and <your-instance-name> with the appropriate values. This command:
- Enumerates all entities whose dataPlatformInstance aspect references the specified platform instance.
- Hard deletes those child entities recursively.
- Removes the platform instance entity itself.
Warning: This permanently deletes all datasets and containers associated with the platform instance. Ensure you intend to remove all of these entities before proceeding.
If entities are already orphaned (hard delete was performed out of order). If you already hard deleted the platform instance entity before running a recursive delete, child entities may be orphaned and unreachable by the standard recursive delete. In this case:
- Attempt the recursive delete command above — it may still enumerate entities via the Elasticsearch index.
- If orphaned entities remain visible in the UI after running the command, contact DataHub Support to request a manual backend cleanup and index refresh for your environment.
Verify removal. After completing either approach, re-run your ingestion pipeline to confirm only the intended platform instances appear. Check the UI filter dropdown under the relevant data platform to confirm the removed instance no longer appears. If the instance still appears briefly, allow time for the Elasticsearch index to refresh.
Correct your ingestion recipes to prevent recurrence. Ensure all related ingestion sources (e.g., dbt, Tableau, downstream BI tools) have consistent and correctly configured platform_instance and platform_instance_map values that align with the target platform's platform_instance setting. Mismatched values across recipes produce duplicate containers and broken lineage.
```
# Example: dbt ingestion aligned to a Redshift platform instance
source:
  type: dbt
  config:
    # ... other configs ...
    target_platform: redshift
    target_platform_instance: <your-redshift-platform-instance>
    platform_instance: <your-dbt-platform-instance>
```

Additional Notes

The datahub delete --recursive flag explicitly supports dataPlatformInstance entity types (alongside container) as a first-class recursive deletion target. This behavior is controlled internally via the _RECURSIVE_DELETE_TYPES set in the DataHub CLI. Stateful ingestion is the strongly preferred long-term solution as it continuously manages entity lifecycle and prevents orphaned entities from accumulating over time. If your ingestion recipe specifies a database parameter, you generally do not need to repeat the database name inside schema_pattern allow rules — doing so can produce unintended duplicate containers. The Tableau embedded datasource lineage behavior (where embedded data sources appear as intermediate nodes before upstream platform tables) is expected; expanding the embedded data source node in the lineage view will reveal the upstream platform tables. A bug causing embedded datasource IDs to leak into published datasource filters was resolved in CLI v1.5.0.1 and DataHub Cloud v1.0.0-cloud.

Issue

You Might Be Asking

Root Cause

Solution

Additional Notes

Related Documentation