Resyncing DataHub Search Indices After Migration Using RestoreIndices – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Search Index Synchronization / Post-Migration Operations

Issue

After completing a large-scale bulk data migration into DataHub, asset counts visible in the search UI may not align with what is stored in the primary SQL database. This discrepancy occurs because high-volume metadata event processing can cause Kafka MCE/MAE events to lag or be dropped, leaving Elasticsearch temporarily out of sync with the SQL store. Individual asset pages may still be reachable by direct URN-based URL, confirming the records exist in SQL but have not been fully indexed into Elasticsearch. The recommended remediation is to run the RestoreIndices job, scoped to targeted entity types using urnLike filters, to reindex Elasticsearch from the SQL database without re-ingesting source data.

You Might Be Asking

Why do my asset counts in the DataHub UI not match what was migrated into the database?
How do I resync Elasticsearch with the SQL store after a migration without re-running ingestion?
Can I run multiple RestoreIndices jobs in parallel to speed up reindexing?
Which entity types should I prioritize, and which ones should not run concurrently?
How do I handle small entity types like Domain, Tag, and TermGroup differently from large datasets?
How do I monitor whether the RestoreIndices job is making progress?

Solution

Use the RestoreIndices operation with urnLike filters to perform a targeted, batched reindex of Elasticsearch from the SQL store. The following approach is recommended for large-scale migration scenarios.

Understand the two execution paths for RestoreIndices.
- Self-serve API (/openapi/operations/elasticSearch/restoreIndices): Suitable for small entity types (e.g., Domain, Tag, TermGroup, GlossaryTerm). These have lower record counts and can be triggered directly by users with admin access.
- Backend Kubernetes job: Required for large entity types (e.g., datasets, containers, pipelines, dashboards). This path bypasses the resource limits of the customer-facing API and is the appropriate method for high-volume reindexing. On DataHub Cloud, this is executed by the DataHub support/operations team.

Design non-overlapping urnLike filters to scope each job.

Breaking the reindex into discrete, non-overlapping batches allows multiple jobs to run in parallel safely. The following filter patterns cover common entity types. Adapt the platform names to match your environment:

# Containers
urnLike=urn:li:container:%

# Glossary terms
urnLike=urn:li:glossaryTerm:%

# Datasets — small-to-medium platforms (can run in parallel with each other)
urnLike=urn:li:dataset:(urn:li:dataPlatform:databricks,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:db2,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:files,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:azuresynapse,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:incorta,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:powerbi,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:greenplum,%

# Datasets — large platforms (run separately, not concurrently with each other)
urnLike=urn:li:dataset:(urn:li:dataPlatform:mssql,%
urnLike=urn:li:dataset:(urn:li:dataPlatform:oracle,%

# Pipelines
urnLike=urn:li:dataFlow:(<platform>,%
urnLike=urn:li:dataJob:(urn:li:dataFlow:(<platform>,%

# BI assets
urnLike=urn:li:dashboard:%
urnLike=urn:li:chart:%

Replace <platform> with the relevant data platform identifier (e.g., mssql, oracle).

Sequence job execution to manage resource contention.
- Run small-to-medium entity type filters in parallel — they are non-overlapping and safe to execute concurrently.
- Run very large entity type filters (e.g., those covering platforms with millions of records) sequentially or in isolation to avoid resource contention on Elasticsearch and the underlying database.
- Prioritize higher-value entity types first (e.g., containers and glossary terms before lower-priority BI assets) if timeline is constrained.
Handle small entity types via the self-serve API.
Entities such as Domain, Tag, TermGroup, and GlossaryTerm have low record counts and can be resynced directly using the documented REST API endpoint. Use the following pattern:
```
POST /openapi/operations/elasticSearch/restoreIndices
Content-Type: application/json

{
  "urnLike": "urn:li:tag:%"
}
```
Repeat for each small entity type as needed. Refer to the RestoreIndices API documentation for full parameter details.
Monitor job progress using Kafka consumer lag.
The primary indicator of RestoreIndices job progress is the Metadata Change Log (MCL) Kafka consumer lag. As the job processes records and emits MCL events, the consumer lag will initially rise and then drain toward zero as Elasticsearch processes the updates. Monitor the MCL consumer group lag for your instance to track completion. On DataHub Cloud, the operations team monitors this metric directly.
Validate asset counts after reindexing completes.
Once all jobs have finished and Kafka consumer lag has returned to near zero, verify asset counts per entity type and per platform in the DataHub UI. Minor residual discrepancies may require further investigation (e.g., filtering by specific platforms or entity types to identify gaps). A spot-check approach — confirming that individual URNs are reachable via direct URL — can help distinguish indexing gaps from data loss.

Additional Notes

This scenario — Elasticsearch drifting out of sync with the SQL store following high-volume bulk migration — is an expected operational artifact, not a product regression. DataHub's event-driven architecture can experience lag or event drops at very high ingestion volumes. RestoreIndices is the supported remediation path for this condition.
On DataHub Cloud, large-scale RestoreIndices jobs that require backend Kubernetes execution must be triggered by the DataHub support or cloud operations team. Customers should open a support ticket and provide their urnLike filters. The self-serve API endpoint is subject to resource limits that make it unsuitable for very large entity volumes.
On self-hosted DataHub, administrators with Kubernetes access can run the RestoreIndices job directly as a backend job. Refer to the official documentation for the job configuration and CLI options.
Use urnBasedPagination when available, as it provides more reliable pagination behavior for large result sets during reindexing operations.
This operation is appropriate for post-migration sync, recovery from Elasticsearch write failures, disaster recovery scenarios where an Elasticsearch cluster was lost, or any situation where Elasticsearch has drifted from the primary SQL store.
Plan for RestoreIndices operations to be a repeatable step in your migration runbook if you are performing staged or multi-phase migrations (e.g., staging environment validation followed by production cutover).
After reindexing, if discrepancies persist for specific entity types, consider running a targeted reindex for just those URN patterns before concluding validation.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation