Risks and Alternatives to Using DataHub-to-DataHub Instance Sync for Temporary Metadata Sharing – DataHub Customer Support Portal

Area: Best Practices

Sub-Area: Multi-Instance Metadata Management

Issue

Organizations running multiple DataHub instances — for example, one per business unit or environment — sometimes consider copying or syncing entities from one instance to another as a temporary bridge while ingestion pipelines are being configured on the destination instance. While DataHub does offer a DataHub-to-DataHub integration feature (visible under Settings → Integrations) designed for controlled metadata sharing, attempting to bulk-copy entities between instances carries significant risks: many entity aspects are instance-specific, computed, or reference objects that do not exist in the destination, leading to broken lineage, phantom siblings, stale usage statistics, and persistent metadata corruption that native ingestion cannot automatically clean up.

You Might Be Asking

Can I use the DataHub-to-DataHub integration in Settings to sync entities between two DataHub Cloud instances?
Is it safe to copy metadata from one DataHub instance to another while I set up ingestion on the destination?
Which entity aspects are safe to copy between DataHub instances?
What happens to computed aspects like usage statistics or browse paths if I copy them from another instance?
What is the recommended approach when two teams need access to the same metadata but run separate DataHub instances?

Solution

The recommended solution is to configure the same ingestion sources independently on each DataHub instance rather than attempting to copy or sync entities between instances. The sections below explain why cross-instance copying is problematic and, if copying is absolutely necessary as a last resort, which aspects are safe to migrate.

Understand the DataHub-to-DataHub integration feature and its current scope.
The DataHub-to-DataHub integration (Settings → Integrations) is a private beta feature designed for selective, opt-in metadata sharing between two DataHub Cloud instances. Key characteristics:
- Unidirectional by design: Sharing is always initiated from one instance to another. Bidirectional sharing requires configuring outbound shares on each side independently.
- Per-entity and opt-in: Users select individual entities to share via the UI or GraphQL API. The entire catalog is not synced by default.
- Lineage control: When sharing an entity, you can choose to include upstream lineage, downstream lineage, both, or neither.
- Supported entity types: Datasets, Containers, Dashboards, Charts, DataJobs, DataFlows, ML Models, Feature Tables, Domains, Data Products, Tags, Glossary Terms, Glossary Nodes, Users, and Groups.
- Automated re-sync: A scheduled job can re-sync previously shared entities on a recurring basis.
This feature is appropriate for ongoing, controlled cross-team visibility — not for bulk migration of an entire catalog from one instance to another.
Understand why bulk entity copying causes metadata integrity problems.
Every DataHub entity is composed of multiple aspects — the fundamental building blocks that store different facets of metadata. When considering copying a dataset entity between instances, aspects fall into three risk categories:

Aspects that are safe to copy (produced by ingestion sources and portable):
- datasetProperties – name, description, custom properties
- schemaMetadata – column and field definitions
- subTypes – further classification
- dataPlatformInstance – platform identifier
Aspects DataHub computes internally (not safe to copy — will be incorrect and conflict with destination-computed values):
- lineageFeatures – upstream/downstream count summaries derived from the source instance's graph
- usageFeatures – query and usage statistics tied to the source instance's query logs
- browsePathsV2 – navigation paths computed from what exists in the source instance
Aspects that reference objects not present in the destination instance (will break or render phantom references):
- siblings – references sibling datasets (e.g., a linked BigQuery dataset) that may not exist in the destination
- testResults – references data quality tests that only exist in the source instance
- upstreamLineage – will render phantom upstream datasets if those upstreams haven't been ingested into the destination
- institutionalMemory – may link to internal documentation inaccessible to users of the destination instance
- ownership, globalTags, glossaryTerms – may reference owners (different user accounts), tags, or terms that do not exist in the destination
Critical risk: When the destination instance later ingests the same sources natively, ingestion will overwrite only the aspects it produces. Aspects written by the copy process — such as usageFeatures, siblings, or testResults — will persist on those entities indefinitely because native ingestion does not clean up aspects written by a different process. This results in stale, incorrect metadata that requires significant manual remediation.
Prefer configuring independent ingestion sources on the destination instance.
The correct long-term solution is to configure the same ingestion sources (dbt, BigQuery, Snowflake, etc.) directly on the destination DataHub instance. This ensures:
- All aspects are computed correctly in the context of the destination instance.
- Lineage, usage, browse paths, and siblings are accurate and self-consistent.
- Native ingestion lifecycle management (soft deletes, aspect cleanup) functions correctly.
- No manual remediation is required after the fact.

If copying is an absolute last resort, copy only safe aspects.

If blocking business requirements make it impossible to wait for ingestion configuration and a temporary copy is unavoidable, restrict the copy strictly to the safe aspects listed above. The following Python snippet illustrates a minimal, targeted approach using the DataHub Python SDK:

from datahub.ingestion.graph.client import DataHubGraph, DataHubGraphConfig
from datahub.emitter.mcp import MetadataChangeProposalWrapper

# Connect to the source instance
source_instance = DataHubGraph(DataHubGraphConfig(
    server="https://<source-instance>.acryl.io/gms",
    token="<source-instance-token>"
))

# Connect to the destination instance
dest_instance = DataHubGraph(DataHubGraphConfig(
    server="https://<destination-instance>.acryl.io/gms",
    token="<destination-instance-token>"
))

# Only copy aspects that are safe to transfer between instances.
# DO NOT include: lineageFeatures, usageFeatures, browsePathsV2,
# siblings, testResults, upstreamLineage, ownership, globalTags,
# glossaryTerms, or institutionalMemory without careful review.
SAFE_ASPECTS = [
    "datasetProperties",
    "schemaMetadata",
    "subTypes",
    "dataPlatformInstance",
]

# Filter by platform and environment as appropriate for your use case
for urn in source_instance.get_urns_by_filter(platform="<platform-name>", env="PROD"):
    aspects = source_instance.get_entity_semityped(urn, aspects=SAFE_ASPECTS)
    for aspect in aspects.values():
        dest_instance.emit(MetadataChangeProposalWrapper(
            entityUrn=urn,
            aspect=aspect
        ))

Replace <source-instance>, <destination-instance>, <source-instance-token>, <destination-instance-token>, and <platform-name> with values appropriate for your environment. Review each additional aspect carefully before adding it to SAFE_ASPECTS.

Plan for cleanup after native ingestion is configured.
If a temporary copy was performed, after native ingestion is fully configured on the destination instance, audit the copied entities for stale or incorrect aspects using the DataHub OpenAPI or GraphQL API. The full list of aspects for any entity type can be inspected via the OpenAPI GET endpoint under the relevant entity type (e.g., "dataset entity").

Additional Notes

The DataHub-to-DataHub integration available in Settings → Integrations is a private beta feature as of the time of this writing and is subject to change. It is designed for selective, user-initiated metadata sharing between instances — not for bulk catalog migration. Bulk copying of entities between instances, even using the Python SDK, requires careful aspect-level analysis and carries a high risk of persistent metadata corruption if computed or instance-specific aspects are included. The effort required to safely execute and subsequently remediate a bulk copy typically exceeds the effort required to configure native ingestion sources on the destination instance. Always prefer independent ingestion configuration as the primary approach.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation