Understanding and Fixing Column Description Propagation Not Triggering for Programmatically Set Lineage – DataHub Customer Support Portal

Area: Product Issues

Sub-Area: Column Description Propagation / Lineage Automation

Issue

When column-level lineage is configured programmatically via the Python SDK, column descriptions from upstream datasets may not automatically propagate to downstream datasets — even though the lineage mapping is correctly set and valid. This occurs because DataHub's column description propagation engine is event-driven: it reacts to write events on a column's DocumentationClass aspect. Descriptions ingested from a source system (stored in SchemaMetadata.fields[].description) do not fire a documentation-change event on their own, so if the propagation service was unavailable or restarting at the time of ingestion, those descriptions may be permanently skipped until a new write event occurs or a manual backfill is performed.

You Might Be Asking

Why did one column's description propagate immediately after a manual UI edit, but all other columns — which have descriptions from ingestion — did not propagate?
Why doesn't setting column-level lineage via Python automatically trigger propagation of existing upstream descriptions?
How do I force column description propagation for columns whose descriptions were written during a single ingestion run that was never picked up?
Is there a bug if propagation works for manually edited columns but not for ingested descriptions?

Root Cause

DataHub's column description propagation is event-driven and distinguishes between two types of column description writes:

Ingested descriptions — written to SchemaMetadata.fields[].description during a source ingestion run. These do not automatically fire a documentation-change event that triggers propagation.
Authored descriptions — written to the DocumentationClass aspect, either via a UI edit or an explicit SDK emission of that aspect. These do fire the documentation-change event and trigger propagation immediately.

If column lineage is established after descriptions have already been ingested, or if the propagation service experienced a gap (such as a restart) at the time of ingestion, no subsequent write event will occur and propagation will not fire retroactively for those columns. This is a known gap: when column lineage is first set, the system does not retroactively propagate pre-existing ingested source descriptions.

Solution

Choose one of the following approaches depending on the number of columns affected:

Option 1 — UI Backfill (Recommended for most cases)

Navigate to your DataHub instance and open the upstream dataset in the UI.
Run a one-time propagation backfill from the dataset's Actions menu or the Propagation settings panel. This triggers the propagation engine to process all datasets with pending lineage mappings across your instance.
Note that the backfill is not scoped to a single table — it will process all datasets in your instance that are waiting on propagation. Allow sufficient time for it to complete.
Alternatively, for a small number of columns, manually open each column's description in the UI, copy the existing description text, and re-save it. This creates a DocumentationClass entry and immediately triggers propagation for that column.

Option 2 — Python SDK (Recommended for bulk columns)

Emit a DocumentationClass MCP (Metadata Change Proposal) for each upstream column whose description you want propagated. This creates the authored-description event that triggers the propagation engine.

import datahub.emitter.mce_builder as builder
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.metadata.schema_classes import DocumentationClass, DocumentationAssociationClass
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter(gms_server="https://<your-instance>.datahubproject.io/api/gms")

upstream_dataset_urn = "urn:li:dataset:(urn:li:dataPlatform:<platform>,<dataset-name>,PROD)"

# Map of column field path to its description text
column_descriptions = {
    "field_name_1": "Description for field 1",
    "field_name_2": "Description for field 2",
    # Add all columns that need propagation
}

for field_path, description_text in column_descriptions.items():
    documentation = DocumentationClass(
        documentations=[
            DocumentationAssociationClass(
                documentation=description_text,
                attribution=None,
            )
        ]
    )
    mcp = MetadataChangeProposalWrapper(
        entityUrn=upstream_dataset_urn,
        aspect=documentation,
        # Target the specific schema field
        # Use the fully qualified field path as needed by your schema
    )
    emitter.emit(mcp)

print("DocumentationClass aspects emitted — propagation will trigger shortly.")

Once DocumentationClass aspects are emitted for each upstream column, the propagation engine will fire and push descriptions to the downstream dataset(s).

Option 3 — Restart the Propagation Service (Self-hosted deployments)

If you manage your own DataHub deployment and the propagation service experienced a gap or restart during ingestion, restarting the propagation service can allow it to catch up on missed events.
After restarting, perform the UI backfill described in Option 1 to ensure all pending descriptions are processed.

Additional Notes

Priority of descriptions: If a downstream dataset's columns already have their own ingested descriptions from a separate source ingestion run, those ingested descriptions take display priority over propagated descriptions. The propagated descriptions will still be stored and visible in the metadata panel, but they will not appear as the primary displayed description.
Lineage correctness is a prerequisite: Propagation requires that fine-grained column-level lineage is correctly established between upstream and downstream columns. Verify that lineage mappings are 1:1 and that column names match exactly (including case sensitivity) before troubleshooting propagation behavior.
Going forward: Once the backfill is complete and the propagation service is healthy, both new UI edits and new ingested description changes on upstream datasets will propagate in real time. A repeated backfill should not be necessary unless the propagation service experiences another gap.
Known limitation: When column-level lineage is first established programmatically, the system does not retroactively propagate pre-existing ingested descriptions. The workarounds above are the recommended paths until this behavior is addressed in a future product release.
This behavior applies to DataHub Cloud and self-hosted DataHub deployments running versions with the event-driven propagation architecture.