Structured Properties Overwritten by Ingestion: Using PATCH Semantics to Prevent Data Loss – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Structured Properties / Metadata Tests / Ingestion Overwrites

Issue

Structured properties set by metadata tests or custom Python SDK scripts are periodically overwritten and lost when ingestion pipelines run. The symptom is that structured properties appear correctly on assets immediately after a metadata test completes, but revert to a previous state (or are removed entirely) after the next ingestion cycle. This occurs because certain ingestion sources and custom transformers emit a full UPSERT of the structuredProperties aspect, replacing the entire set of property values rather than merging only the fields they manage. As a result, any structured properties written by metadata tests or independent scripts are silently discarded on every ingestion run.

Error Messages

No visible error — structured properties are silently removed or reset after each ingestion run.

You Might Be Asking

Why do structured properties set by my metadata test disappear the next day?
Why does my metadata test show passing results but the structured property value is not applied (or keeps resetting)?
How can I prevent ingestion from overwriting structured properties that were set by a metadata test or SDK script?
What is the difference between PATCH and UPSERT for structured property aspects in DataHub?
How do I configure the set_structured_property metadata test action correctly?

Solution

The fix requires two complementary steps: (1) ensure metadata test actions use the correct YAML structure, and (2) convert any ingestion sources that perform full UPSERT writes of the structuredProperties aspect to use PATCH semantics instead.

Verify the metadata test action YAML is correctly structured.
A known bug in the DataHub UI test builder (fixed in SaaS v2.0.0-cloud / v1.2.0) can produce malformed YAML for set_structured_property actions. Confirm your test definition matches the following format exactly. Note that structured properties only support STRING and NUMBER value types — there is no native Boolean type, so boolean-like values must be stored as strings ('True' / 'False').
```
actions:
  passing:
    - type: set_structured_property
      values:
        - 'urn:li:structuredProperty:<your-property-urn>'
      stringValues:
        - 'True'
  failing:
    - type: set_structured_property
      values:
        - 'urn:li:structuredProperty:<your-property-urn>'
      stringValues:
        - 'False'
```
Common mistakes that cause silent failures:
- Using booleanValues — this key does not exist; use stringValues.
- Omitting the values key that contains the structured property URN.
- Placing the property value directly under values instead of under stringValues or numberValues.
- Placing the action under the wrong condition block (passing vs. failing).
Identify which ingestion source is overwriting structured properties.
Review the timeline of aspect changes for an affected dataset by inspecting its metadata history in the DataHub UI or via the Timeseries Aspect API. Look for writes to the structuredProperties aspect that occur on a regular schedule and reset properties to a subset of values. Common culprits include:
- Custom Python SDK scripts that call emit_mce_async or emit_mcp with a full structuredProperties aspect payload.
- Ingestion recipes that use a simple_add_dataset_domain_structured_properties transformer or similar custom transformers that emit the full aspect.
- Any pipeline that constructs a StructuredPropertiesClass object and emits it without using the patch builder.

Convert UPSERT writes to PATCH operations in Python SDK scripts.

Replace any code that emits a full structuredProperties aspect with the DataHub patch builder. The patch builder emits only the specific property being modified, leaving all other structured properties untouched.

Before (full UPSERT — overwrites all structured properties):

from datahub.metadata.schema_classes import (
    StructuredPropertiesClass,
    StructuredPropertyValueAssignmentClass,
    StringTypeClass,
)

# This replaces ALL structured properties on the entity
mcp = MetadataChangeProposalWrapper(
    entityUrn="urn:li:dataset:(<your-platform>,<your-dataset>,PROD)",
    aspect=StructuredPropertiesClass(
        properties=[
            StructuredPropertyValueAssignmentClass(
                propertyUrn="urn:li:structuredProperty:<your-property-urn>",
                values=["<your-value>"],
            )
        ]
    ),
)
emitter.emit(mcp)

After (PATCH — only modifies the specified property, preserves all others):

from datahub.specific.dataset import DatasetPatchBuilder

patch_builder = DatasetPatchBuilder(
    urn="urn:li:dataset:(<your-platform>,<your-dataset>,PROD)"
)
patch_builder.set_structured_property(
    urn="urn:li:structuredProperty:<your-property-urn>",
    value="<your-value>",
)

for mcp in patch_builder.build():
    emitter.emit(mcp)

A full working example is available in the DataHub open-source repository at metadata-ingestion/examples/library/dataset_add_structured_properties_patch.py.

Convert UPSERT transformer configurations to PATCH in ingestion recipes.

If you are using a transformer such as simple_add_dataset_domain_structured_properties in your recipe, verify whether that transformer emits a full aspect or uses patch semantics. If it performs a full UPSERT, either replace it with a patch-capable equivalent or remove it from the recipe and manage those properties via a separate PATCH script.

# Example ingestion recipe snippet — replace full-UPSERT transformer
# with a patch-based approach or remove if properties are managed elsewhere

source:
  type: bigquery
  config:
    # ... your BigQuery config ...

# Remove or replace transformers that overwrite the full structuredProperties aspect:
# transformers:
#   - type: simple_add_dataset_domain_structured_properties  # UPSERT — avoid this
#     config:
#       properties:
#         <your-property-key>: '<your-property-value>'

Validate the fix.
After updating scripts and/or transformers:
- Trigger a manual ingestion run and then re-run the metadata test.
- Navigate to an affected dataset and confirm that all structured properties — both those set by ingestion and those set by the metadata test — are present simultaneously.
- Monitor the structured property values across at least two full ingestion + test cycles to confirm stability.

Additional Notes

UI bug in test builder (set_structured_property actions): Versions of DataHub Cloud prior to SaaS v2.0.0-cloud / v1.2.0 contain a bug where the UI test builder incorrectly serializes set_structured_property action YAML, causing the action to silently fail at runtime. If you are on an affected version, you must manually edit the test YAML to match the correct structure shown above. This bug is resolved in SaaS v2.0.0-cloud.

No native Boolean type: DataHub structured properties support only STRING, RICH_TEXT, NUMBER, DATE, and URN logical value types. To represent boolean values, declare the structured property as STRING type and use string values such as 'True' and 'False'.

Silent failure mode: When a metadata test action fails at runtime (for example, due to a type mismatch), the test engine catches the exception internally. The test results UI may show the test as having run successfully while the action was never applied. Check GMS application logs for entries containing Caught exception while attempting to apply action or Failed to apply metadata test action to diagnose silent failures.

Siblings / combined views: If an entity has sibling assets (for example, a BigQuery table and a corresponding dbt model), structured properties may differ between siblings. The combined (stitched) entity view in the UI displays a merged badge that can reflect the property from the non-primary sibling. Verify the property value on each sibling asset individually if the badge appears incorrect.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation