Bulk Clearing User-Edited Descriptions to Reveal Ingested Source Descriptions – DataHub Customer Support Portal

Area: Best Practices

Sub-Area: Description Management & Dual-Aspect Architecture

Issue

When DataHub ingests descriptions from a source system (such as Snowflake), those descriptions are stored in ingestion-managed aspects (datasetProperties for tables, schemaMetadata for columns). If a user has also manually edited a description in the DataHub UI, that edit is stored in a separate, parallel aspect (editableDatasetProperties for tables, editableSchemaMetadata for columns). The UI always displays the user-edited value first, effectively masking the ingested source description. When teams want to revert to showing the ingested descriptions — for example, after a source system's descriptions have been improved or synchronized — they need a way to bulk-clear the user-edited aspect values without triggering a full re-ingestion and without inadvertently wiping tags, glossary terms, or columns that have no backing ingested description.

You Might Be Asking

Why isn't the description from my Snowflake ingestion showing up even after I re-run the ingestion pipeline?
How do I make DataHub show the ingested description instead of the manually edited one?
Is there a way to bulk-reset user-edited descriptions across many datasets at once?
Will clearing editable descriptions also delete my column tags and glossary terms?
Do I need to re-ingest my source to see updated descriptions after clearing user edits?

Solution

The resolution involves selectively patching the editable aspects to remove only the description field, leaving all other user-curated metadata (tags, glossary terms, etc.) intact. The steps below walk through the architecture, the key caveats, and a safe scripted approach with dry-run support.

Step 1 — Understand the Dual-Aspect Architecture

DataHub maintains two separate sets of aspects for descriptions:

Ingestion-managed aspects (datasetProperties, schemaMetadata): Written by ingestion pipelines and updated on each ingestion run.
User-editable aspects (editableDatasetProperties, editableSchemaMetadata): Written when a user edits a description in the UI. These always take display priority over ingested values.

No re-ingestion is required to reveal the ingested description. Removing or patching the user-editable aspect is sufficient; the ingested value will appear immediately.

Step 2 — Review Critical Caveats Before Running Any Script

Tags and glossary terms share the editable aspect. A naive full deletion of editableSchemaMetadata will also delete all user-applied column tags and glossary terms on that dataset. Always patch (update) rather than delete the aspect so only the description field is cleared.
Column operations are all-or-nothing per dataset. The editableSchemaMetadata aspect covers all columns in a dataset. Patching it requires care to update only the target column descriptions while leaving all other columns untouched.
No ingested description = blank result. If you clear a user-edited description on an asset where the source system has no description, the field will become blank. Identify and skip these assets before running a bulk operation.
Always test in a non-production environment first and use a dry-run mode before applying changes to production.

Step 3 — Prepare Your URN List

Collect the URNs for the datasets and columns you want to process. You can export these from the DataHub UI or query the API. Before running the script, review each URN to confirm the source system actually has a description behind the user-edited one; skip any assets where clearing would result in a blank description.

Step 4 — Run the Bulk Clear Script

The following Python script patches the editable description to an empty string (or removes it) for each URN you provide, while preserving all other metadata. It defaults to a dry-run mode — pass --apply to make real changes. It also writes a backup of the current state before applying any modifications.

"""
clear_editable_descriptions.py

Bulk-clears user-edited descriptions from DataHub editable aspects
while preserving tags, glossary terms, and all other metadata.

Usage:
  # Dry run (default — no changes made):
  python clear_editable_descriptions.py

  # Apply changes to a limited set first:
  python clear_editable_descriptions.py --apply --limit 5

  # Apply changes to all URNs in the list:
  python clear_editable_descriptions.py --apply

Requirements:
  pip install requests
"""

import argparse
import json
import requests
from datetime import datetime

# ---------------------------------------------------------------
# CONFIGURATION — update these values for your environment
# ---------------------------------------------------------------
DATAHUB_URL = "https://<your-instance>.acryl.io"  # e.g. https://your-company.acryl.io
ACCESS_TOKEN = "<your-personal-access-token>"      # Settings → Access Tokens

# List of dataset URNs to process.
# For column-level edits, the script reads editableSchemaMetadata
# from each dataset URN and patches only the description fields.
URNS = [
    "urn:li:dataset:(urn:li:dataPlatform:snowflake,<your_database>.<your_schema>.<your_table>,PROD)",
    # Add additional URNs here...
]
# ---------------------------------------------------------------

HEADERS = {
    "Authorization": f"Bearer {ACCESS_TOKEN}",
    "Content-Type": "application/json",
}

BACKUP_FILE = f"description_backup_{datetime.utcnow().strftime('%Y%m%dT%H%M%SZ')}.json"


def get_aspect(urn: str, aspect_name: str) -> dict | None:
    """Fetch a single aspect for a URN via the OpenAPI v3 endpoint."""
    encoded_urn = requests.utils.quote(urn, safe="")
    url = f"{DATAHUB_URL}/openapi/v3/entity/dataset/{encoded_urn}/aspects/{aspect_name}"
    response = requests.get(url, headers=HEADERS)
    if response.status_code == 404:
        return None  # Aspect does not exist — nothing to clear
    response.raise_for_status()
    return response.json()


def patch_editable_dataset_properties(urn: str, dry_run: bool) -> dict:
    """Clear the table-level editable description, preserving all other fields."""
    aspect = get_aspect(urn, "editableDatasetProperties")
    if aspect is None:
        return {"urn": urn, "aspect": "editableDatasetProperties", "status": "skipped_no_aspect"}

    current_description = aspect.get("value", {}).get("description", "")
    if not current_description:
        return {"urn": urn, "aspect": "editableDatasetProperties", "status": "skipped_already_empty"}

    patched = dict(aspect.get("value", {}))
    patched["description"] = ""

    if dry_run:
        return {
            "urn": urn,
            "aspect": "editableDatasetProperties",
            "status": "dry_run",
            "would_clear": current_description,
        }

    encoded_urn = requests.utils.quote(urn, safe="")
    url = f"{DATAHUB_URL}/openapi/v3/entity/dataset/{encoded_urn}/aspects/editableDatasetProperties"
    response = requests.post(url, headers=HEADERS, json={"value": patched})
    response.raise_for_status()
    return {
        "urn": urn,
        "aspect": "editableDatasetProperties",
        "status": "cleared",
        "previous_description": current_description,
    }


def patch_editable_schema_metadata(urn: str, dry_run: bool) -> dict:
    """
    Clear user-edited column descriptions while preserving tags,
    glossary terms, and any columns that have no ingested description
    behind them.

    NOTE: This example clears ALL column descriptions in the editable
    aspect. Extend the logic to filter specific columns or to check
    for a backing ingested description before clearing.
    """
    aspect = get_aspect(urn, "editableSchemaMetadata")
    if aspect is None:
        return {"urn": urn, "aspect": "editableSchemaMetadata", "status": "skipped_no_aspect"}

    fields = aspect.get("value", {}).get("editableSchemaFieldInfo", [])
    if not fields:
        return {"urn": urn, "aspect": "editableSchemaMetadata", "status": "skipped_no_fields"}

    changes = []
    patched_fields = []
    for field in fields:
        field_copy = dict(field)
        description = field_copy.get("description", "")
        if description:
            # TODO: Add a check here to confirm the source system has a
            # backing description before clearing; skip if it would go blank.
            changes.append({"field_path": field_copy.get("fieldPath"), "cleared": description})
            field_copy["description"] = ""
        patched_fields.append(field_copy)

    if not changes:
        return {"urn": urn, "aspect": "editableSchemaMetadata", "status": "skipped_no_descriptions"}

    if dry_run:
        return {
            "urn": urn,
            "aspect": "editableSchemaMetadata",
            "status": "dry_run",
            "would_clear": changes,
        }

    patched_value = dict(aspect.get("value", {}))
    patched_value["editableSchemaFieldInfo"] = patched_fields

    encoded_urn = requests.utils.quote(urn, safe="")
    url = f"{DATAHUB_URL}/openapi/v3/entity/dataset/{encoded_urn}/aspects/editableSchemaMetadata"
    response = requests.post(url, headers=HEADERS, json={"value": patched_value})
    response.raise_for_status()
    return {
        "urn": urn,
        "aspect": "editableSchemaMetadata",
        "status": "cleared",
        "fields_cleared": changes,
    }


def save_backup(backups: list) -> None:
    with open(BACKUP_FILE, "w") as f:
        json.dump(backups, f, indent=2)
    print(f"Backup saved to {BACKUP_FILE}")


def main():
    parser = argparse.ArgumentParser(description="Clear user-edited DataHub descriptions.")
    parser.add_argument("--apply", action="store_true", help="Apply changes (default is dry-run).")
    parser.add_argument("--limit", type=int, default=None, help="Process only the first N URNs.")
    args = parser.parse_args()

    dry_run = not args.apply
    urns = URNS[: args.limit] if args.limit else URNS

    if dry_run:
        print("DRY RUN MODE — no changes will be made. Pass --apply to commit.")
    else:
        print(f"APPLY MODE — changes will be written to {DATAHUB_URL}")

    backups = []
    results = []

    for urn in urns:
        # Back up current state before any changes
        for aspect_name in ("editableDatasetProperties", "editableSchemaMetadata"):
            current = get_aspect(urn, aspect_name)
            if current:
                backups.append({"urn": urn, "aspect": aspect_name, "snapshot": current})

        results.append(patch_editable_dataset_properties(urn, dry_run))
        results.append(patch_editable_schema_metadata(urn, dry_run))

    if not dry_run:
        save_backup(backups)

    print("\n=== RESULTS ===")
    for r in results:
        print(json.dumps(r, indent=2))


if __name__ == "__main__":
    main()

Step 5 — Recommended Execution Order

Install the dependency:
```
pip install requests
```
Populate the URNS list and set DATAHUB_URL and ACCESS_TOKEN in the script.
Run a dry run and review the output report:
```
python clear_editable_descriptions.py
```
Apply to a small batch first to verify results in the UI:
```
python clear_editable_descriptions.py --apply --limit 5
```
If the sample looks correct, apply to all URNs:
```
python clear_editable_descriptions.py --apply
```
Confirm in the DataHub UI that the ingested source descriptions are now visible. No re-ingestion is required.

Additional Notes

This behavior — user-edited descriptions taking display priority over ingested descriptions — is intentional, long-standing DataHub design. The separation of ingestion-managed aspects from user-editable aspects protects human curation from being silently overwritten by automated ingestion runs. This architecture has been stable since DataHub v0.9.x and is not a bug or regression. When extending the script for production use, it is strongly recommended to add pre-flight logic that checks whether the ingested aspect contains a non-empty description before clearing the user-edited value; this prevents assets from being left with a blank description field. The backup file written by the script can be used to restore previous values via the same OpenAPI endpoint if needed. Always test changes in a non-production environment before running against production assets.