Outdated Lineage Edges Not Being Removed

Stale lineage relationships persisted in DataHub even after the underlying data pipelines or table relationships had changed or been removed. The lineage edge removal mechanism was not properly identifying and cleaning up outdated lineage connections, leading to incorrect or misleading lineage graphs.

You Might Be Asking:

• Why do I still see old lineage connections?
• How does DataHub handle lineage cleanup?
• How can I force remove stale lineage?

The lineage edge removal logic has been fixed to properly identify and remove outdated relationships. The system now: - Tracks lineage edge timestamps more accurately - Compares current vs. historical lineage states - Removes edges that are no longer reported by ingestion sources

For immediate cleanup of stale lineage, re-run your ingestion sources with the stateful_ingestion feature enabled:

source:
  type: <your-source>
  config:
    # Your source config
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

Fixed in version v0.3.12. Stateful ingestion must be enabled for automatic cleanup of stale lineage. Manual lineage edges created through the UI are not affected by this cleanup process.

• Lineage Feature Guide
• Stateful Ingestion