Skip to main content

Column-Level Structured Properties Validation in Metadata Tests

Angelo Bresci
Updated

Area: Product Issues

Sub-Area: Metadata Tests

Issue

When creating metadata tests for column-level structured properties through the DataHub UI, the test validation fails with an error indicating that the generated query format is invalid for dataset entities. This occurs because the UI generates an incorrect property path format that the backend query evaluator cannot process.

Error Messages

  • Failed to validate test definition: Query schemaFieldStructuredProperties.urn:li:structuredProperty:[property-id] is invalid for entity type dataset: No eligible query evaluator found. Make sure the schemaFieldStructuredProperties is a valid aspect

You Might Be Asking

  • Why do column-level structured property tests fail validation when created through the UI?
  • How can I test structured properties on schema fields in metadata tests?
  • What operators are supported for column-level structured property tests?

Solution

This issue was caused by a frontend-backend mismatch in query format generation and has been resolved in DataHub Cloud v0.3.18. The fix includes:

  1. Corrected Property Path Format: The UI now generates the correct property path format that the backend expects: 
    property: 'schemaFields.structuredProperties.urn:li:structuredProperty:[property-id]'
    Instead of the previous incorrect format: 
    property: 'schemaFieldStructuredProperties.urn:li:structuredProperty:[property-id]'
  2. Operator Limitation: Column-level structured property tests now only support the exists operator in the UI, as other operators (equals, contains, etc.) are not reliably supported for this property type.

After the fix, you can create metadata tests for column-level structured properties by:

  1. Navigating to the Metadata Tests section in the DataHub UI
  2. Creating a new test with entity type dataset
  3. Adding conditions and rules that reference column-level structured properties
  4. Using only the exists operator for structured property validations

Example working test configuration:

on:
  types:
    - dataset
conditions:
  and:
    - property: dataPlatformInstance.platform
      operator: equals
      values:
        - 'urn:li:dataPlatform:oracle'
rules:
  and:
    - property: 'schemaFields.structuredProperties.urn:li:structuredProperty:[your-property-id]'
      operator: exists
actions:
  failing:
    - type: add_tags
      values:
        - 'urn:li:tag:missing_property'

Additional Notes

This was a product bug that affected the initial release of column-level structured property support in metadata tests. The issue has been resolved in DataHub Cloud v0.3.18 and later versions. If you encounter this issue on older versions, contact your DataHub support team for upgrade guidance. Currently, only the exists operator is supported for column-level structured property tests due to technical limitations in how these properties are evaluated across schema fields.

Related Documentation

Tags: metadata-tests, structured-properties, validation-error, column-level, schema-fields, product-bug, query-evaluator, frontend-backend, exists-operator, dataset-entity