Skip to main content

Resolving PATCH vs UPSERT Change Type Issues in CLI

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Change Type Configuration

Issue

When using DataHub CLI version 1.4.0+ with Hive Metastore ingestion, the generated MCPs show "changeType": "PATCH" for datasetProperties instead of the expected "UPSERT". This behavior affects ingestion workflows that expect full replacement semantics rather than merge operations.

You Might Be Asking

  • How can I force UPSERT instead of PATCH for datasetProperties?
  • Why did my ingestion change from UPSERT to PATCH after upgrading CLI?
  • What's the difference between PATCH and UPSERT change types?

Solution

The PATCH change type for datasetProperties is controlled by the enable_properties_merge flag, which defaults to true in DataHub CLI v0.13.2.3 and later versions.

  1. To revert to UPSERT semantics, add the following configuration to your Hive Metastore recipe: 
    source:
  type: hive-metastore
  config:
    enable_properties_merge: false
    # ... your existing config
  2. If you encounter GraphQL endpoint issues when lineage features are enabled, add a separate datahub_api section: 
    datahub_api:
  server: "https://:8080"
  token: ""

source:
  type: hive-metastore
  config:
    enable_properties_merge: false
    include_view_lineage: true
    include_column_lineage: true
    # ... rest of your config

sink:
  type: datahub-rest
  config:
    server: "https://.datahubproject.io"
  3. Alternatively, disable lineage features if you don't need them: 
    source:
  type: hive-metastore
  config:
    enable_properties_merge: false
    include_view_lineage: false
    include_column_lineage: false
    include_view_column_lineage: false

Additional Notes

Be aware that using UPSERT semantics will cause each ingestion run to fully replace the datasetProperties aspect. This means any custom properties not present in the Hive source (such as properties added manually or by separate ingestion jobs) will be removed on the next run. If you rely on properties from multiple sources or manual enrichments, consider keeping the PATCH behavior. The container aspect reporting issue mentioned in some cases was resolved in CLI version 1.5.0.13+.

Related Documentation

Tags: patch, upsert, hive-metastore, change-type, cli, properties-merge, ingestion-configuration, datasetproperties, mcp

Related articles