Structured Property Hard Delete Blocks Recreation Due to Persistent Elasticsearch Field Mapping – DataHub Customer Support Portal

Area: Product Issues

Sub-Area: Structured Properties / Elasticsearch Index Management

Issue

When a structured property is hard-deleted and a user attempts to recreate a property with the same qualified name, DataHub rejects the creation request with an error stating that the Elasticsearch field derived from that qualified name already exists in the entity-index mapping. This occurs because Elasticsearch does not support dropping fields from an existing index mapping — once a field is added, it persists even after the structured property definition is removed from DataHub. Because DataHub normalizes qualified names by replacing . with _ (e.g., global.env becomes global_env), the residual mapping blocks any new property whose qualified name normalizes to the same field name. The only way to clear the residual field is to rebuild the affected entity indices via the SystemUpdate job.

Error Messages

Structured property Elasticsearch field 'global_env' (from qualifiedName='global.env') already exists in the entity-index mapping. This is either a live collision with another property whose qualified name normalizes to the same field ('.' and '_' both normalize to '_'), or a residual mapping left by a hard-deleted property with this field name (Elasticsearch cannot drop a field from a mapping, so the field survives the delete). To resolve: choose a qualified name that stays distinct after '.'->'_', or, when recreating a deleted property, complete its assignment cleanup and reindex the entity indices (via the SystemUpdate job) before recreating it.

You Might Be Asking

Why can't I recreate a structured property after deleting it?
Why does DataHub block a new structured property that has a unique qualified name?
How do I clear a residual Elasticsearch field left by a hard-deleted structured property?
Can I resolve this without contacting DataHub support (self-service)?
What is the difference between a hard delete and a soft delete for structured properties?

Solution

There are two distinct root causes for this error. Identify which applies before proceeding.

Determine whether this is a live collision or a residual mapping.
- Live collision: A currently active (or soft-deleted) structured property exists whose qualified name also normalizes to the same Elasticsearch field. In this case, choose a different qualified name that remains distinct after . → _ normalization. A reindex will not resolve a live collision.
- Residual mapping (hard delete): The original structured property was hard-deleted. The definition is gone but the Elasticsearch field persists. This is the scenario addressed by the steps below.
Confirm the property is fully hard-deleted.
If the property is only soft-deleted, the Elasticsearch field will not be removed during reindex. Ensure the definition has been fully removed before proceeding.
Remove all residual property assignments.
Verify that no entities still carry a value for the deleted structured property in their structuredProperties aspect. Lingering assignments on entities can prevent the field from being cleared from the index during rebuild and may cause additional errors such as:
```
no valid property assignments remain after removing values for non-existent properties
```
Clean up any remaining assignments before triggering the reindex.
Trigger a SystemUpdate reindex of the affected entity indices.
The SystemUpdate job detects removed structured-property fields and rebuilds entity indices from the current property definitions only. Ensure the following flags are enabled in your deployment configuration:
```
datahubSystemUpdate:
  enableStructuredProperties: true

global:
  elasticsearch:
    index:
      enableMappingsReindex: true
```
With these flags set, run the SystemUpdate job against your DataHub instance. The job clones existing indices before rebuilding, so the risk of data loss is low. Note that large indices may take significant time to rebuild.

For DataHub Cloud deployments, this reindex must be initiated by DataHub support — it cannot be triggered from the UI or public API. Contact DataHub support and reference this article.
Verify the field has been removed from all affected indices.
The structured property field is added to the index mapping for every entity type declared in the property's definition — not only entity types that had actual values assigned. Confirm the field has been cleared from all applicable entity indices (e.g., dataset, domain, glossaryTerm, glossaryNode, etc.) before attempting recreation.

You can inspect the current Elasticsearch mapping for a given index using:
```
GET /<your-index-name>/_mapping
```
Confirm the field (e.g., global_env) no longer appears under the structuredProperties mapping for any relevant index.
Recreate the structured property.
Once the reindex has completed and the field is confirmed removed from all affected indices, recreate the structured property with the original qualified name via the UI or API.

Recommended Alternative: Avoid Hard Delete

To avoid this situation entirely, prefer the following approaches over hard deletion:

Soft delete the property instead of hard-deleting it. A soft-deleted property retains its definition in DataHub, so its Elasticsearch field name remains reserved and no collision occurs upon recreation.
Bump the property version when making breaking schema changes. Incrementing the version field (recommended format: yyyyMMddhhmmss) keeps the same URN and qualified name, preserving the Elasticsearch field without triggering this conflict.

Example of updating a structured property with a version bump via the DataHub Python SDK:

from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph

graph = DataHubGraph(DatahubClientConfig(server="https://<your-instance>.datahubproject.io"))

# Bump the version when making breaking changes to avoid Elasticsearch field conflicts
structured_property_patch = {
    "version": "20241015120000",  # yyyyMMddhhmmss format
    # ... other property fields ...
}

See the official documentation on updating structured properties with breaking schema changes for the full recommended workflow.

Additional Notes

This behavior is intentional as of DataHub version v2.2.0. Prior to v2.2, DataHub did not validate for field name collisions at property creation time, meaning a hard-delete-and-recreate workflow would succeed silently but could cause type conflicts or stale values in search results. The validation was introduced to surface this pre-existing limitation explicitly.
The root cause is a fundamental Elasticsearch constraint: fields can be added to an index mapping but cannot be removed without rebuilding the index. This is not a DataHub defect.
Qualified name normalization means that global.env and global_env produce identical Elasticsearch field names (global_env). If you need to use a different qualified name to avoid this conflict, ensure the new name is distinct after applying the . → _ substitution.
If your automation pipeline hard-deletes and recreates structured properties on a recurring schedule, each hard-delete cycle will block the name again until a reindex is performed. Migrate such workflows to use soft delete or version bumping instead.
For DataHub Cloud deployments, the SystemUpdate reindex job is not user-accessible. Submit a support request to have support staff trigger the rebuild on your behalf.
This issue is tracked internally. Automatic cleanup of Elasticsearch fields upon hard delete of a structured property is a known open item and may be addressed in a future release.