Stateful Ingestion Not Removing Assets Created by Lineage – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Stateful Ingestion and Cross-Source Entity Management

Issue

Temporary tables or other assets appear in DataHub despite no longer existing in the source system, even when stateful ingestion is properly configured with remove_stale_metadata: true. These assets often show recent "last observed" timestamps but have no associated run IDs in their system metadata, indicating they were created or refreshed by a different ingestion source than the one configured for stateful cleanup.

You Might Be Asking

Why are expired BigQuery temporary tables still showing in DataHub when stateful ingestion is enabled?
Why don't these assets have run IDs in their system metadata despite appearing recently?
How can lineage ingestion interfere with metadata ingestion's stateful cleanup?

Solution

This issue occurs when assets are created or refreshed by multiple ingestion sources (commonly lineage/usage sources), but only one source has stateful ingestion enabled. Stateful ingestion can only remove entities that were originally created by the same source.

Identify the source creating the assets:
Check if you have separate ingestion sources running (e.g., BigQuery metadata source and BigQuery lineage/usage source). The lineage source often creates entities when it processes audit logs that reference tables in queries, including expired temporary tables.
Configure filtering on all sources:
Add table filtering to prevent unwanted assets from being ingested by any source. For BigQuery temporary tables with expiration timestamps, add this to your recipe:
```
source:
  type: bigquery
  config:
    table_pattern:
      deny:
        - ".*_expires_.*"  # Matches tables with expiration timestamps
        - ".*temp.*"       # Matches other temporary table patterns
    # ... rest of your config
```
Apply this same filtering to both your metadata ingestion source and lineage/usage source recipes.

Verify stateful ingestion configuration:

Ensure your main metadata source has proper stateful ingestion enabled:

source:
  type: bigquery
  config:
    # ... your existing config
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
      fail_safe_threshold: 75.0  # Increase if you expect many deletions

Clean up existing stale assets:
For assets already in DataHub that won't be cleaned up automatically, you have two options:
- UI Method: Navigate to the dataset in DataHub UI, click the "..." menu, and select "Delete" to soft-delete the assets
- API Method: Use the batch delete API for bulk removal:
```
curl -X POST '.datahubproject.io/api/graphql' \
  -H 'Authorization: Bearer ' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "mutation { batchUpdateSoftDeleted(input: { urns: [\"\"], deleted: true }) }"
  }'
```
Monitor future ingestions:
After implementing the filtering, run your ingestion sources and verify that new temporary tables are not being created and existing filtered tables are removed in subsequent runs.

Additional Notes

This is a common architectural limitation where cross-source entity ownership can prevent proper cleanup. The lineage source may continue refreshing entities even after the metadata source removes them from its scope. Always ensure consistent filtering across all ingestion sources that might interact with the same datasets. For BigQuery specifically, the default temp_table_dataset_prefix filter only applies to dataset names starting with underscore, not to table-level naming patterns within regular datasets.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation