Data Asset Count Mismatches During Migration from DataHub Core to DataHub Cloud – DataHub Customer Support Portal

```html

Area: Deployment Issues

Sub-Area: Core-to-Cloud Metadata Migration

Issue

When migrating metadata from a self-hosted DataHub Core instance to DataHub Cloud using file-based export and import, administrators frequently observe discrepancies between the total asset count shown for a platform and the counts shown in the Tables, Views, or other subtype breakdowns. Soft-deleted entities may also appear in browse and search results when they should be hidden, and container hierarchies may render incorrectly. These symptoms share several root causes: missing subTypes aspects on migrated datasets, URN format differences between the export and the Cloud environment, multiple chunked import runs inadvertently soft-deleting each other's entities through stateful ingestion, and the Elasticsearch/OpenSearch index falling out of sync with the primary metadata store after large-scale writes.

Error Messages

Platform total count does not match sum of Table + View breakdowns in the UI
Soft-deleted entities visible in browse/search after migration
Container hierarchy missing child schemas or databases in the Browse UI
Search facet shows N datasets but no results appear when that facet is selected

You Might Be Asking

Why does the platform total show 473 k datasets but the Tables + Views breakdown only adds up to a fraction of that?
Why are soft-deleted entities from my Core instance appearing in search and browse on Cloud?
Why are some containers missing from the Browse hierarchy even though their child datasets are visible?
Will re-indexing Elasticsearch fix the count mismatches, or is something wrong with the imported data itself?
My chunked file upload used different pipeline names per file — can stateful ingestion still cause soft-deletes across files?
Do I need to sort my chunk files so that container aspects load before dataset aspects?

Root Causes

Understanding which root cause applies to your environment is essential before choosing a remediation strategy. The following causes are not mutually exclusive and commonly occur together in large migrations:

Missing subTypes aspect on migrated datasets. DataHub uses the subTypes aspect to classify a dataset as a Table, View, Stream, etc. Datasets that were exported without this aspect — either because the original connector never wrote it, or because it was lost during transformation — count toward the platform total but do not appear in any subtype bucket. The totals are internally consistent; the breakdown simply cannot account for unclassified entities.
Stateful ingestion soft-deleting entities across chunked import runs. When a large metadata file is split into chunks and each chunk is uploaded as a separate pipeline run, any pipeline name that overlaps with a prior run triggers stateful stale-entity removal. Entities present in an earlier chunk but absent from a later chunk's pipeline scope are marked removed = true. This produces a characteristic pattern: active entities trace back to early import runs while soft-deleted entities concentrate in later ones. Even with different pipeline names per file, implicit stateful ingestion activation (e.g., via the presence of a pipeline_name key) can trigger this behavior.
URN format differences between Core and Cloud. If the export, transformation scripts, or native connector recipes produce URNs in different casing or with different platform-instance segments than what was written during original ingestion, the Cloud instance treats them as new entities. Fully enriched entities from earlier connector runs (with subTypes, schemaMetadata, and container aspects) are soft-deleted because they are no longer "seen," and minimal file-imported duplicates take their place.
Elasticsearch/OpenSearch index out of sync with the primary metadata store. Large-scale file imports can cause the search index to lag behind the database or to retain stale documents. Specifically: (a) soft-deleted entities can lose their removed flag in the index if a later import pass overwrites the index document with a minimal payload that omits the flag; (b) hard-deleting live container URNs does not cascade into already-soft-deleted child entities, leaving orphaned index documents that inflate facet counts; and (c) orphaned index documents with no corresponding database row appear as ghost entries in search facets.
Soft-deleted legacy entities included in the export. If the DataHub source connector is run with include_soft_deleted_entities: true (or without explicitly setting it to false), entities that were previously cleaned up in the Core instance are exported and re-imported into Cloud, artificially inflating counts and polluting the browse hierarchy.

Solution

Phase 1 — Diagnose Which Root Causes Are Present

Check whether the total equals the sum of subtypes plus unclassified entities. If Tables + Views + (no subtype) equals the platform total, the index is consistent and the issue is missing subTypes aspects, not a sync problem. You can verify a specific dataset's aspects via GraphQL:
```
query CheckSubTypes {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:oracle,<your-dataset>,PROD)") {
    urn
    subTypes {
      typeNames
    }
    status {
      removed
    }
    container {
      urn
    }
  }
}
```
Verify whether soft-deleted entities are leaking into search. Check a suspect entity's status in both the database and the index. If status.removed is true in the database but the entity is visible in browse or search, the index document is out of sync:
```
query CheckSoftDelete {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:<platform>,<dataset-name>,PROD)") {
    urn
    status {
      removed
    }
  }
}
```

Verify whether container browse paths are present. A container that is hidden despite having active child datasets is typically soft-deleted itself, not missing browse-path metadata:

query CheckContainer {
  container(urn: "urn:li:container:<container-guid>") {
    urn
    properties { name }
    status { removed }
    browsePathV2 {
      path {
        name
        entity { urn }
      }
    }
  }
}

Inspect import run IDs for the stateful-deletion pattern. In DataHub Cloud, contact Support to pull systemMetadata.runId for a sample of active and soft-deleted entities within the same platform. If active entities consistently trace to earlier run IDs and soft-deleted entities trace to later run IDs within the same import job, multiple chunked runs with overlapping stateful ingestion scope are the cause.

Phase 2 — Remediate Elasticsearch / Index Sync Issues

In DataHub Cloud, customers cannot directly access or reindex the Elasticsearch/OpenSearch backend. Contact DataHub Support to request an index restore for affected platforms. An index restore rebuilds search documents from the primary metadata store, which correctly re-applies removed = true flags and removes stale documents. After a restore completes, allow additional time (up to 30–60 minutes for large platforms) for all UI facets and counts to reflect the updated index before re-validating counts.

To remove pure index ghosts (documents with no backing database row), DataHub Support must perform a targeted index-side cleanup, as these entities cannot be reached by the CLI datahub delete command.

Phase 3 — Reset the Environment and Re-run the Migration Cleanly

If the migration data itself is corrupted by soft-delete cross-contamination or URN mismatches, the most reliable remediation is a controlled environment reset followed by a single coherent import with the issues corrected. This approach preserves curated metadata (tags, glossary terms, ownership, descriptions) that a from-scratch native connector re-crawl would not.

Capture baseline counts from the Core instance before resetting. Record the per-platform dataset count, container count, and total aspect row count. This is the reconciliation target for the reload.
Hard-delete active entities by platform from the Cloud environment. Run the following CLI command once per platform. Use a Kubernetes job or equivalent to avoid a single long-lived connection timing out on large platforms:
```
datahub delete --hard --platform <platform-name>
```
Repeat for each platform in scope. This command does not affect policies, secrets, service accounts, users, groups, or ingestion sources, as those are not tied to a data platform.

Important: A platform-scoped hard delete is driven by the search index and cannot reach entities that are already soft-deleted in the database. Soft-deleted entities from prior runs remain and must be purged separately (see next step).
Purge residual soft-deleted entities. After the platform deletes complete, run:
```
datahub delete --hard --only-soft-deleted --dry-run
```
Review the output, then run without --dry-run to remove the soft-deleted tombstones from both the database and the index. Batch this job the same way as the platform deletes if the volume is large.
Clean up orphaned non-dataset entity types. Platform-scoped deletes do not remove assertions, queries, tests, or incidents that were associated with the deleted datasets. These are real database entities that become orphaned. Remove them by entity type if a clean baseline is required:
```
# Dry-run first for each type
datahub delete --entity-type assertion --hard --dry-run
datahub delete --entity-type query --hard --dry-run
datahub delete --entity-type incident --hard --dry-run

# Then remove the --dry-run flag to execute
datahub delete --entity-type assertion --hard
datahub delete --entity-type query --hard
datahub delete --entity-type incident --hard
```
Warning: --entity-type assertion --hard removes all assertions in the environment. Confirm scope before running.
Request backend verification from DataHub Support. Before uploading, ask Support to confirm that both the metadata database and the search index are at zero for all target platforms. CLI commands only scan datasets and cannot confirm index cleanliness for all entity types or detect orphaned index documents.

Update the export recipe to exclude soft-deleted entities and system metadata. Ensure the DataHub source connector recipe includes the following:

source:
  type: datahub
  config:
    include_soft_deleted_entities: false
    include_all_versions: false
    stateful_ingestion:
      enabled: true
      ignore_old_state: false
    urn_pattern:
      deny:
        - ^urn:li:dataHubRole.*
        - ^urn:li:dataHubPolicy.*
        - ^urn:li:dataHubIngestionSource.*
        - ^urn:li:dataHubSecret.*
        - ^urn:li:dataHubExecutionRequest.*
        - ^urn:li:dataHubAccessToken.*
        - ^urn:li:dataHubUpgrade.*
        - ^urn:li:inviteToken.*
        - ^urn:li:globalSettings.*
        - ^urn:li:dataHubStepState.*
        - ^urn:li:corpuser.*
        - ^urn:li:corpGroup.*

flags:
  set_system_metadata: false

Disable stateful ingestion on the file upload (import) pipeline. Each chunk file must be uploaded with stateful ingestion explicitly disabled to prevent later chunks from soft-deleting entities loaded by earlier chunks:
```
pipeline_config = {
    "pipeline_name": "metadata-migration-<chunk-sequence>",
    "source": {
        "type": "file",
        "config": {
            "path": "<path-to-chunk-file>",
            "stateful_ingestion": {
                "enabled": False
            }
        }
    },
    "sink": <your-sink-config>
}
```
Each chunk should use a unique pipeline_name (e.g., include the file sequence number) to prevent any cross-chunk stateful state from accumulating.
Verify URN consistency before uploading. Confirm that the URN format produced by the transformation scripts (casing, platform instance segment, environment suffix) matches exactly what the native connectors produce for the same entities in the Cloud environment. A mismatch causes the file-imported entity and the connector-ingested entity to be treated as separate assets, with the connector run soft-deleting the file-imported version.
Validate after the reload. The migration is complete when all of the following are true:
- Per-platform dataset count reconciles to the Core baseline count.
- Tables + Views + (no subtype) equals the platform total, with (no subtype) at or near zero.
- A sample URN per platform carries subTypes, container, and schemaMetadata aspects.
- The browse hierarchy renders correctly for a sample database and schema.
- No soft-deleted entities appear in browse or search.
- Kafka consumer lag for the GMS consumer group has reached zero.

Phase 4 — Address Missing subTypes via Native Connector Re-ingestion

If datasets were originally ingested into Core through native connectors that do emit subTypes, those aspects should be present in the Core database. Verify this before assuming re-ingestion is needed:

Confirm the subTypes aspect exists on the entities in the Core backend. If Core never wrote it, no import strategy will produce it, and native connector re-ingestion against the Cloud environment is the only path to populating subtype classifications.
If running native connectors against Cloud, ensure the connector URN conventions (casing, platform instance, environment) are aligned with the migrated URNs before running, to avoid creating duplicate entities.

Additional Notes

DataHub Cloud managed infrastructure: In DataHub Cloud, the Elasticsearch/OpenSearch backend is fully managed by Acryl. Customers cannot directly reindex or modify the search backend. All index restore and orphaned-document cleanup operations must be performed by DataHub Support.

Chunk file ordering: Loading child entities (datasets) before their parent containers is safe. DataHub does not perform referential integrity checks at ingest time that would drop a child aspect because its parent container has not yet been written. The parent