Entity Visible via URN but Missing from Container Browse View Due to Elasticsearch Index Drift – DataHub Customer Support Portal

```html

Area: Product Issues

Sub-Area: Search & Browse / Elasticsearch Index Consistency

Issue

A dataset entity (e.g., a BigQuery table) is accessible via its direct URN in DataHub and all relevant graph-store aspects — container, status, and browsePathsV2 — are confirmed correct via the REST API, yet the entity does not appear in its parent container's Contents view. Re-running ingestion multiple times does not resolve the problem. This is caused by Elasticsearch index drift: the search/browse index is out of sync with the authoritative SQL-backed graph store for that specific entity. A secondary, related symptom is that a dbt metadata model and its corresponding warehouse table entity may appear as two separate entities rather than being correctly paired as siblings, while other tables in the same container pair correctly.

Error Messages

POST /openapi/v1/operations?action=restoreIndices — 403 Forbidden (insufficient privileges)
Entity not found in container Contents view despite correct container aspect in graph store

You Might Be Asking

Why is an entity accessible by direct URN but not visible in the container browse panel?
Why didn't re-running ingestion fix the missing entity in the container Contents view?
Why is my dbt model showing as a separate entity instead of being paired with its BigQuery table?
Why does POST /operations?action=restoreIndices return a 403 error?
Could there be other entities silently missing from browse views without my knowledge?

Solution

Issue 1: Entity Missing from Container Browse View (Elasticsearch Index Drift)

Confirm the root cause. Verify that the graph-store aspects are correct by querying the REST API directly. If all three aspects below are correct but the entity is still absent from the container Contents view, the problem is a stale or missing Elasticsearch document for that entity.

# Check the container aspect
GET /api/v2/aspects/{urlEncodedUrn}?aspect=container&version=0

# Expected response indicates the correct parent container URN:
{
  "aspect": {
    "com.linkedin.container.Container": {
      "container": "urn:li:container:<parent-container-urn>"
    }
  }
}

# Check the status aspect — entity must not be soft-deleted
GET /api/v2/aspects/{urlEncodedUrn}?aspect=status&version=0

# Expected:
{
  "aspect": {
    "com.linkedin.common.Status": {
      "removed": false
    }
  }
}

# Check the browsePathsV2 aspect
GET /api/v2/aspects/{urlEncodedUrn}?aspect=browsePathsV2&version=0

# Expected path includes parent container URNs in order:
{
  "aspect": {
    "com.linkedin.common.BrowsePathsV2": {
      "path": [
        {"urn": "urn:li:container:<grandparent-container-urn>"},
        {"urn": "urn:li:container:<parent-container-urn>"}
      ]
    }
  }
}

Understand why re-running ingestion does not fix it. The ingestion pipeline emits metadata events that update the graph store. However, if the Elasticsearch indexing step for a specific entity was silently skipped or failed during a prior run (e.g., due to a transient error), subsequent ingestion runs that detect no change in the aspect values will not re-emit the Elasticsearch update. This results in a persistent gap between the graph store and the search index for that entity.
Trigger a targeted Elasticsearch re-index for the affected entity. The restoreIndices operation re-reads the aspect data from the SQL graph store and re-emits the Elasticsearch document, closing the gap.
- DataHub Cloud customers: The restoreIndices endpoint requires the RESTORE_INDICES_PRIVILEGE policy, which is not granted to standard customer API tokens. Contact DataHub Support and provide the affected entity URN. Support engineers can run the operation on your behalf using the internal OpenAPI endpoint:
```
curl -sS -X POST 'https://<your-instance>.datahubproject.io/openapi/operations/elasticSearch/restoreIndices' \
  -H "Authorization: Bearer $DATAHUB_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '["urn:li:dataset:(urn:li:dataPlatform:<platform>,<fully-qualified-table-name>,PROD)"]'
```
  A successful response will look like:
```
[
  {
    "ignored": 0,
    "rowsMigrated": 11,
    "timeSqlQueryMs": 5,
    "timeGetRowMs": 0,
    "lastUrn": "urn:li:dataset:(urn:li:dataPlatform:<platform>,<fully-qualified-table-name>,PROD)",
    "lastAspect": "browsePathsV2"
  }
]
```
- Self-hosted DataHub customers: If your user or service account has the RESTORE_INDICES_PRIVILEGE granted, you can call the endpoint directly as shown above, or use the DataHub Admin UI under Settings > Administration > Manage Indices if available in your version.
Consider a broader audit for silent data gaps. If this drift occurred for one entity, it may have occurred for others. You can request (or run, if self-hosted with appropriate access) a broader health check that scans for entities where the graph-store container aspect exists but the Elasticsearch document's container field is absent or mismatched. On DataHub Cloud, contact Support to request an environment-wide index consistency audit.

Issue 2: dbt Model and Warehouse Table Appearing as Separate Entities

DataHub pairs dbt models with their corresponding warehouse entities (e.g., BigQuery tables) by constructing sibling relationships at ingest time. The matching is entirely URN-driven. If the URN produced by the dbt ingestion source does not exactly match the URN produced by the warehouse ingestion source, no sibling relationship is written and the two entities appear separately. Importantly, a re-index will not fix this — the sibling aspect itself is absent from the graph store and must be established through corrected ingestion.

Check target_platform_instance alignment. In your dbt ingestion recipe, the target_platform_instance value must exactly match the platform_instance value used in your warehouse (e.g., BigQuery) ingestion recipe. If one is set and the other is not, or if the values differ, pairing will silently fail.

# dbt ingestion recipe (datahub-ingestion config)
source:
  type: dbt
  config:
    target_platform: bigquery
    target_platform_instance: <your-platform-instance>   # Must match BigQuery recipe exactly
    # ... other config

# BigQuery ingestion recipe
source:
  type: bigquery
  config:
    platform_instance: <your-platform-instance>           # Must match dbt recipe exactly
    # ... other config

Check convert_urns_to_lowercase consistency. Both the dbt and warehouse ingestion recipes must use the same casing normalization setting. A mismatch produces URNs that appear identical but differ in case, breaking the sibling relationship silently.
```
# Ensure this setting is identical in both recipes:
source:
  config:
    convert_urns_to_lowercase: true   # or false — must be the same in both
```
Verify the dbt artifact files are current. If a specific model is missing or stale in the dbt manifest.json or catalog.json artifact files, the dbt ingestion source will not build a complete model node for it, and the sibling aspect will never be written. Check that the affected model is present and fully built in both artifact files. If stale, regenerate the artifacts with a fresh dbt run and dbt docs generate, then re-run the dbt ingestion pipeline.
Re-run both pipelines with consistent configuration. After aligning the above settings, re-run both the warehouse ingestion pipeline and the dbt ingestion pipeline. The sibling relationship will be established on the next successful run.
If pairing still fails after config alignment, contact DataHub Support and request a direct inspection of the sibling aspects for the affected entities via the REST API. Support can identify which side of the pair has the URN mismatch.

Additional Notes

Elasticsearch drift vs. product regression: The scenario where an entity is accessible by direct URN but absent from browse/search is a known, long-standing category of eventual-consistency drift in DataHub. It is not specific to any single release. Because DataHub uses a SQL-backed graph store as the source of truth and Elasticsearch as a derived search index, transient indexing failures during ingestion can cause isolated gaps. The fact that the vast majority of entities in a given ingestion run are indexed correctly while only one or two are missing is characteristic of an isolated drift event rather than a systemic regression.

Authorization note: The RESTORE_INDICES_PRIVILEGE policy privilege is intentionally restricted. On DataHub Cloud, this operation is performed by Support engineers. On self-hosted deployments, a DataHub administrator with the appropriate policy grant can invoke the endpoint directly.

Scope of restoreIndices: The operation can be scoped to a single URN (as shown above) or to a specific list of aspect names to minimize impact. For large-scale drift, a broader restore can be triggered but should be coordinated with your infrastructure team, as it places additional load on Elasticsearch.

Version applicability: This behavior has been observed across multiple DataHub versions in the v0.3.x line and earlier. The dbt/warehouse sibling pairing mechanism and the target_platform_instance matching logic have been stable across recent releases.