Dataset Deprecation and Soft Deletion Issues

Datasets marked as deprecated or soft-deleted are still appearing in search results or recommendations, or the deprecation status is not persisting. This typically occurs when the deprecation aspect is not properly set, search indices haven't updated, or stateful ingestion is removing the deprecation flag.

You Might Be Asking:

• How do I mark a dataset as deprecated?
• Why do deprecated datasets still appear in search?
• How do I permanently delete a dataset?

1. Mark dataset as deprecated via GraphQL:

mutation updateDeprecation {
  updateDeprecation(
    input: {
      urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.table,PROD)"
      deprecated: true
      note: "This table is deprecated. Use analytics.new_table instead."
      decommissionTime: 1735689600000  # Unix timestamp for when to decommission
    }
  )
}

2. Set deprecation via Python SDK:

Example for Snowflake. This can be adapted for other data sources/connectors.

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.metadata.schema_classes import DeprecationClass, AuditStampClass
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter("http://localhost:8080")

deprecation = DeprecationClass(
    deprecated=True,
    note="Replaced by new_table",
    actor="urn:li:corpuser:datahub",
    decommissionTime=1735689600000  # Optional: When to fully remove
)

emitter.emit_mcp(
    entity_urn=make_dataset_urn("snowflake", "db.schema.table"),
    aspect_name="deprecation",
    aspect=deprecation
)

3. Filter deprecated datasets in search:

query searchActiveDatasets {
  search(
    input: {
      type: DATASET
      query: "*"
      filters: [
        { field: "deprecated", values: ["false"] }
      ]
    }
  ) {
    searchResults {
      entity { urn }
    }
  }
}

4. Prevent ingestion from overwriting deprecation:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    # Don't override manual deprecation status
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: false  # Don't auto-remove

    # Or use transformers to preserve deprecation
    transformers:
      - type: "set_dataset_status"
        config:
          # Only mark as deprecated if missing from source
          # Don't un-deprecate manually deprecated datasets

5. Hard delete (remove entirely from DataHub):

Example for Snowflake. This can be adapted for other data sources/connectors.

# Using CLI
datahub delete --urn "urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.table,PROD)"

# Via API
curl -X POST http://localhost:8080/entities?action=delete \
  -H "Content-Type: application/json" \
  -d '{"urn": "urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.table,PROD)"}'

Deprecation is a soft delete - the asset remains searchable but is marked as deprecated. Hard delete permanently removes the entity from DataHub. Use deprecation for end-of-life planning and hard delete only when necessary.

• Dataset Deprecation
• Deleting Metadata