Custom Properties Not Displaying

Custom properties added to datasets or other entities are not appearing in the DataHub UI or are being overwritten by ingestion. This happens when custom properties are not properly formatted, the UI view doesn't support them, or ingestion overwrites them without preserving existing values.

You Might Be Asking:

• How do I add custom properties to a dataset?
• Where do custom properties appear in the UI?
• Can I prevent ingestion from overwriting custom properties?

1. Add custom properties via API:

Example for Snowflake. This can be adapted for other data sources/connectors.

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import DatasetPropertiesClass

emitter = DatahubRestEmitter("http://localhost:8080")

dataset_properties = DatasetPropertiesClass(
    customProperties={
        "data_classification": "confidential",
        "retention_period": "7_years",
        "business_unit": "finance",
        "cost_center": "CC-12345"
    }
)

emitter.emit_mcp(
    entity_urn="urn:li:dataset:(urn:li:dataPlatform:snowflake,db.schema.table,PROD)",
    aspect_name="datasetProperties",
    aspect=dataset_properties
)

2. View custom properties in the UI:

   • Navigate to dataset → Properties tab
   • Custom properties appear in the "Custom Properties" section
   • May require page refresh after emission

3. Preserve custom properties during ingestion:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    # Add custom properties without overwriting
    transformers:
      - type: "add_dataset_properties"
        config:
          add_properties:
            source_system: "snowflake"
            ingestion_date: "2025-01-15"
          # Set to false to merge with existing properties
          replace_existing: false

4. Bulk add custom properties using transformers:

transformers:
  - type: "pattern_add_dataset_properties"
    config:
      semantics: PATCH  # Don't override existing
      properties_pattern:
        rules:
          "finance.*":
            department: "Finance"
            sensitivity: "high"
          "public.*":
            department: "Public"
            sensitivity: "low"

5. Use GraphQL to query custom properties:

query {
  dataset(urn: "YOUR_URN") {
    properties {
      customProperties {
        key
        value
      }
    }
  }
}

Custom properties are key-value pairs that can store any metadata not covered by standard DataHub fields. Consider using Structured Properties for more strongly-typed custom metadata with validation.

• Custom Properties
• Dataset Properties