Lineage Removal via API Not Immediately Reflected in UI Due to Backend Caching – DataHub Customer Support Portal

Area: Product Issues

Sub-Area: Lineage — Caching and UI Consistency

Issue

When programmatically removing lineage relationships in DataHub — either by updating the UpstreamLineage aspect via the Python SDK or by calling the updateLineage GraphQL mutation — the deletion is correctly applied at the metadata store level but may not be immediately reflected in the DataHub UI or in searchAcrossLineage GraphQL API responses. This occurs because the UI and the lineage search API use a backend cache that is intentionally set to serve cached results (skipCache: false) for performance reasons. Even after confirming that the upstream entry has been removed from the aspect, the stale lineage edge can continue to appear in the UI and in API responses for up to 24 hours while the cache remains warm.

Error Messages

searchAcrossLineage returns degree=1 for a removed upstream/downstream relationship

You Might Be Asking

Is updating the UpstreamLineage aspect and calling the updateLineage mutation the correct way to remove dataset and column-level lineage?
Why does the DataHub UI still show a lineage edge that I have already deleted programmatically?
How can I verify that lineage was actually removed if the UI still displays it?
Is there a way to force a lineage cache refresh or immediately reflect deletions in the UI?
Will using both the SDK aspect update and the GraphQL mutation together cause conflicts?

Solution

Confirm the correct removal approach.

Both of the following methods are valid and supported for removing lineage. You do not need to use both simultaneously — choose one approach:

Option A — Update the UpstreamLineage aspect via the Python SDK:

Fetch the current aspect for the downstream entity.
Remove the target upstream entry from the upstreams list and remove any related entries from fineGrainedLineages (column-level lineage).
Emit the updated aspect.

# Step 1: Fetch the current UpstreamLineage aspect
upstream_lineage = graph.get_aspect(
    entity_urn="<downstream-dataset-urn>",
    aspect_type=UpstreamLineageClass
)

# Step 2: Modify — remove the target upstream and any fine-grained lineage entries
upstream_lineage.upstreams = [
    u for u in upstream_lineage.upstreams
    if u.dataset != "<upstream-dataset-urn>"
]
upstream_lineage.fineGrainedLineages = [
    fg for fg in (upstream_lineage.fineGrainedLineages or [])
    if "<upstream-dataset-urn>" not in (fg.upstreams or [])
]

# Step 3: Emit the updated aspect
mcp = MetadataChangeProposalWrapper(
    entityUrn="<downstream-dataset-urn>",
    aspect=upstream_lineage,
)
datahub_rest_emitter.emit(mcp)

Option B — Use the updateLineage GraphQL mutation:

mutation UpdateLineage($input: UpdateLineageInput!) {
  updateLineage(input: $input)
}

# Variables:
{
  "input": {
    "edgesToAdd": [],
    "edgesToRemove": [
      {
        "upstreamUrn": "<upstream-dataset-urn>",
        "downstreamUrn": "<downstream-dataset-urn>"
      }
    ]
  }
}

Note: DataHub documentation states that lineage added by hand and programmatically may conflict with one another. Avoid combining both methods on the same entity in rapid succession, as this can produce unintended side effects.

Verify the deletion at the API level using skipCache: true.
To confirm the lineage edge has been removed without waiting for the cache to expire, pass skipCache: true in your searchAcrossLineage GraphQL query. This bypasses the backend cache and returns the live state of the lineage graph.
```
query GetDownstreamLineage($urn: String!) {
  searchAcrossLineage(input: {
    urn: $urn,
    direction: DOWNSTREAM,
    start: 0,
    count: 100,
    skipCache: true
  }) {
    searchResults {
      degree
      entity {
        urn
      }
    }
  }
}
```
If the removed entity no longer appears in the results when skipCache: true is set, the deletion has been successfully applied at the backend. The UI will reflect this change once the cache expires (up to 24 hours).
Understand the UI caching behavior.
The DataHub UI intentionally sets skipCache: false on all lineage tab queries for performance reasons, since lineage reads are far more frequent than lineage writes. This is a deliberate trade-off. As a result, lineage deletions may not appear in the UI immediately after the backend is updated.

For most use cases where lineage deletion is an infrequent operation, the recommended approach is to:
- Use skipCache: true in programmatic/API verification workflows to confirm the deletion was applied.
- Allow the UI cache to expire naturally (up to ~24 hours) for the UI to reflect the change.
(DataHub Cloud only) Request a cache disable or forced reindex if needed.
In DataHub Cloud environments, a feature flag exists that can disable the searchAcrossLineage cache globally, causing every lineage page load to fetch live data. Be aware that enabling this flag will result in slower UI performance on the lineage tab. If your use case requires lineage deletions to appear immediately in the UI on a frequent basis, contact DataHub Support to discuss enabling this option. For infrequent deletion workflows, it is generally preferable to leave the cache enabled.

If lineage still appears after more than 24 hours even with cache bypass confirmed to show the deletion, contact DataHub Support to request a forced Elasticsearch reindex.

Additional Notes

The backend cache for lineage queries is a known architectural trade-off in DataHub. The UpstreamLineage aspect is the authoritative source of truth for lineage relationships. The searchAcrossLineage API and the UI lineage graph are derived views that rely on indexed and cached data. Always use skipCache: true in automated pipelines or scripts that need to immediately verify the effect of a lineage write or delete operation. Mixing the SDK aspect update approach and the updateLineage GraphQL mutation on the same entity is not recommended, as the two methods may conflict. Choose one method and apply it consistently. This behavior applies to both DataHub Open Source and DataHub Cloud deployments, though cache TTL and reindex controls may differ between the two.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation