Metadata Model Extension and Custom Entities

Organizations need to add custom metadata fields, create new entity types, or extend existing DataHub entities beyond the standard schema. This requires understanding the metadata model and proper schema extension procedures.

You Might Be Asking:

• Can I add custom fields to datasets?
• How do I create a new entity type?
• What's the difference between custom properties and structured properties?

1. Use Structured Properties for typed custom metadata:

# Create a structured property definition
mutation createStructuredProperty {
  createStructuredProperty(
    input: {
      id: "io.company.dataClassification"
      qualifiedName: "Data Classification"
      valueType: "STRING"
      description: "Internal data classification level"
      allowedValues: [
        {value: "public", description: "Public data"}
        {value: "internal", description: "Internal use only"}
        {value: "confidential", description: "Confidential"}
        {value: "restricted", description: "Highly restricted"}
      ]
      entityTypes: ["dataset", "chart", "dashboard"]
    }
  )
}

2. Apply structured properties to entities:

Example for Snowflake. This can be adapted for other data sources/connectors.

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.metadata.schema_classes import StructuredPropertiesClass

structured_props = StructuredPropertiesClass(
    properties=[
        StructuredPropertyValueAssignmentClass(
            propertyUrn="urn:li:structuredProperty:io.company.dataClassification",
            values=["confidential"]
        ),
        StructuredPropertyValueAssignmentClass(
            propertyUrn="urn:li:structuredProperty:io.company.retentionPeriod",
            values=["7_years"]
        )
    ]
)

emitter.emit_mcp(
    entity_urn=make_dataset_urn("snowflake", "db.schema.table"),
    aspect_name="structuredProperties",
    aspect=structured_props
)

3. For simple key-value pairs, use custom properties:

Example for Snowflake. This can be adapted for other data sources/connectors.

from datahub.metadata.schema_classes import DatasetPropertiesClass

dataset_props = DatasetPropertiesClass(
    customProperties={
        "data_source": "salesforce",
        "refresh_frequency": "hourly",
        "owner_team": "analytics"
    }
)

emitter.emit_mcp(
    entity_urn=make_dataset_urn("snowflake", "db.schema.table"),
    aspect_name="datasetProperties",
    aspect=dataset_props
)

4. Create custom entity types (advanced):

# Define new entity in PDL schema
namespace com.company.metadata

/**
 * A custom ML Model entity
 */
@Entity = {
  "name": "mlModel",
  "keyAspect": "mlModelKey"
}
record MLModel {
  /**
   * Key for the ML Model
   */
  key: MLModelKey

  /**
   * Properties of the ML Model
   */
  properties: MLModelProperties

  /**
   * Ownership information
   */
  ownership: Ownership
}

5. Query custom metadata:

query {
  dataset(urn: "YOUR_URN") {
    # Custom properties
    properties {
      customProperties {
        key
        value
      }
    }

    # Structured properties
    structuredProperties {
      properties {
        propertyUrn
        values
      }
    }
  }
}

Structured Properties provide type safety, validation, and better UI integration compared to custom properties. Creating entirely new entity types requires schema changes and redeployment. For most use cases, structured properties or custom properties are sufficient.

• Structured Properties
• Metadata Model