Structured Property Recreation Fails After Hard Delete Due to Residual Elasticsearch Mapping – DataHub Customer Support Portal

Area: Product Issues

Sub-Area: Structured Properties / Elasticsearch Index Mapping

Issue

When a structured property is hard-deleted in DataHub and then a user attempts to recreate a property with the same qualifiedName, the creation is rejected on every write path. This happens because Elasticsearch/OpenSearch does not support dropping fields from an existing index mapping, the field name remains in the entity-index mapping even after the structured property entity and all its value assignments have been fully removed. Starting in DataHub v2.2, a validator explicitly checks for this collision and blocks recreation of any structured property whose Elasticsearch field name already exists in any entity index mapping, preventing mapping conflicts. The only resolution is to rebuild the affected entity indices via the SystemUpdate job so the stale field mapping is cleared before the property is recreated.

Error Messages

Structured property Elasticsearch field '<qualifiedName>' (from qualifiedName='<qualifiedName>') already exists in the entity-index mapping.
Cannot restrict values that were previously allowed
no valid property assignments remain after removing values for non-existent properties: [urn:li:structuredProperty:<qualifiedName>]

You Might Be Asking

Why does recreating a structured property with the same qualifiedName fail even after I successfully soft-deleted and hard-deleted it?
Why is the Elasticsearch field still present after a hard delete of the structured property?
How do I unblock recreation of a structured property that was hard-deleted?
Does hard-deleting a structured property clean up the Elasticsearch index mapping?
Is there a safer alternative to deleting a structured property when I need to change its allowed values?

Solution

Understanding the Root Cause

Elasticsearch index mappings are append-only. When a structured property is created, DataHub writes a field (e.g., structuredProperties.<qualifiedName>) into the mapping of every entity index associated with that property's definition — not just indices for entities that actually carry values. Hard-deleting the structured property removes the DataHub entity record and cleans up value assignments on carrying entities, but the Elasticsearch field mapping cannot be removed in place. The stale mapping persists until the affected indices are fully rebuilt.

Additionally, note that the stale mapping may exist in more indices than expected. The mapping is written to every entity type declared in the property definition, not just those that held values. For example, a property defined on domains and dataFlows may also have mappings in dataJobIndex and dataProductIndex. The validator scans all of them, so a partial rebuild will not unblock recreation.

Important: Check for Upstream Writers Before Rebuilding

Before triggering a reindex, verify that no pipelines or automation are still writing values for the deleted structured property. After a hard delete, any upstream system that continues to write the old property will re-populate values in the primary store. These values do not appear in the search index (because no definition exists), making them invisible to cleanup, but they will immediately re-surface the moment the property is recreated and the index is rebuilt. If upstream writers are still active, sequence the reindex after those writes have stopped to avoid being out of compliance with the new definition from the first minute.

Step 1: Trigger a Reindex via the SystemUpdate Job (Cloud-Managed Environments)

In DataHub Cloud, customers cannot directly trigger a reindex. Contact DataHub Support to have the operations team run the datahub-system-update job with the following environment variables:

ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true
ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true

These flags trigger the PropertyDefinitionsStep, which re-emits MCLs for all structured property definitions and causes UpdateIndicesHook to rebuild entity index mappings, clearing the residual field. With the default cloneIndices: true setting, the original index is cloned as a backup before rebuilding, so there is no data loss risk.

Step 2: Trigger a Reindex via the SystemUpdate Job (Self-Hosted Environments)

For self-hosted deployments, patch the datahub-system-update Kubernetes job with the required environment variables and trigger a run:

# Example: patch the system-update job via kubectl
kubectl set env job/datahub-system-update \
  ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true \
  ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true \
  -n <your-namespace>

Monitor the job logs to confirm the reindex completes successfully. Verify that the rebuilt index mappings no longer contain the stale structured property field and that document counts match pre-reindex values. Example verification using Elasticsearch APIs:

# Check that the stale field is no longer present in the mapping
GET <your-es-host>/<entity-index>_v2/_mapping

# Confirm document counts match expected values
GET <your-es-host>/<entity-index>_v2/_count

Step 3: Recreate the Structured Property

After the reindex completes and the stale mapping is confirmed removed, recreate the structured property using the CLI, OpenAPI, or SDK with the new definition:

# Example: recreate via DataHub CLI
datahub structured_properties upsert --file structured_property_definition.yaml

# structured_property_definition.yaml example
id: "<your-qualified-name>"
qualified_name: "<your-qualified-name>"
display_name: "<Your Display Name>"
type: "STRING"
entity_types:
  - "urn:li:entityType:datahub.domain"
  - "urn:li:entityType:datahub.dataFlow"
allowed_values:
  - value: "<value1>"
    description: "<description1>"
  - value: "<value2>"
    description: "<description2>"
  - value: "<value3>"
    description: "<description3>"

Alternative: Use Versioning Instead of Deleting (Recommended for Future Changes)

To avoid this situation in the future, consider using the versioning approach instead of deleting and recreating the property. Setting a version field on the structured property definition allows you to restrict allowed values without burning the qualified name.

# Example: upsert with a version string (exactly 14 digits, no dots)
id: "<your-qualified-name>"
qualified_name: "<your-qualified-name>"
version: "20260930120000"
display_name: "<Your Display Name>"
type: "STRING"
allowed_values:
  - value: "<value1>"
  - value: "<value2>"
  - value: "<value3>"

Caveats of the versioning approach:

A version bump renames the underlying Elasticsearch field, so existing property values remain accessible via the UI and API but disappear from search, filters, and aggregations until a restore operation is performed by the DataHub team.
Restricting allowed values via versioning is only possible through the CLI, OpenAPI, or SDK. The UI and GraphQL API cannot narrow allowed values under any circumstances, with or without a version bump.

Additional Notes

This behavior was introduced in DataHub v2.2, where the validator (PropertyDefinitionValidator.java > checkCrossEntityElasticsearchFieldCollision()) was added to explicitly block recreation of a structured property whose Elasticsearch field name already exists in any entity index mapping. This is intentional to prevent mapping conflicts. In Cloud-managed environments, customers cannot trigger the SystemUpdate reindex job directly and must contact DataHub Support to have the operation performed. The reindex rebuilds mappings from existing data with no data loss risk when cloneIndices: true (the default). After a hard delete, any entities that still have stale structured property values written to the primary store by upstream systems will not appear in the index (because no definition exists) but will re-surface immediately once the property is recreated, ensure upstream writers are quiesced before rebuilding to avoid immediate definition violations.