DataHub UI Slowness Caused by Excessive Ingestion Volume and Elasticsearch Index Fragmentation – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Elasticsearch Performance / Ingestion Scope

Issue

DataHub users may experience UI slowness specifically on lineage traversal pages and Discover entity detail pages, while other areas of the UI remain responsive. This pattern is characteristic of Elasticsearch-layer degradation rather than a frontend or application bug. The root causes are typically two compounding factors: (1) index fragmentation in the graph service index caused by high write churn from broad metadata test evaluations, and (2) oversized secondary indices — particularly the query index — accumulating documents without a retention policy and consuming excessive JVM heap, triggering garbage collection pauses that stall queries across the cluster. A general entity cleanup will not resolve this; targeted index maintenance and ingestion scope reduction are required.

Error Messages

point_in_time_time_in_millis: <high cumulative value> — elevated PIT query latency on the graph index
indexing.index_failed: <large count> — indexing failures on dataset or query indices
failed to assign an IP address to container — Kubernetes CNI IP exhaustion from stale ReplicaSets
write rejections: <count> — Elasticsearch node rejecting writes due to heap pressure

You Might Be Asking

Why is lineage loading slowly when the Elasticsearch cluster reports green health?
Will deleting or reducing the number of ingested entities fix the UI slowness?
Why did performance degrade shortly after an Elasticsearch forcemerge that initially improved things?
Why does the graph index keep re-fragmenting after cleanup?
How do metadata tests contribute to lineage index size?

Solution

Step 1: Diagnose Whether Fragmentation Is the Root Cause

A cluster can report green top-level health while individual indices are severely fragmented. Check the following index-level metrics for the graph and query indices:

Query shard-level statistics for the graph index:

GET <your-namespace>_graph_service_v1_*/_stats?filter_path=indices.*.primaries.docs,indices.*.primaries.store,indices.*.primaries.search,indices.*.primaries.indexing

Look for a high deleted-document ratio. A ratio above ~10–15% deleted documents relative to live documents indicates significant fragmentation:
```
# Concerning pattern:
docs.count:   21,000,000
docs.deleted:  3,700,000   # ~17.6% — performance threshold exceeded
```
Check node-level heap usage. Nodes sustaining above 75% heap utilization with Old GC times in the hundreds of thousands of milliseconds will cause stop-the-world pauses that degrade all query latency:
```
GET _cat/nodes?v&h=name,heap.percent,cpu,gc.collectors.old.collection_time
```
Inspect the query index size. An unexpectedly large query index (tens of GB, tens of millions of documents) suggests missing retention:
```
GET _cat/indices/<your-namespace>_queryindex_v2*?v&s=store.size:desc
```

Step 2: Identify the Write Source Behind Ongoing Fragmentation

If a forcemerge was already performed but fragmentation returned within days, the index is being continuously re-written by an upstream process. The most common cause in this pattern is metadata tests evaluating against a very large entity scope, generating one relationship edge per dataset per failing test on every nightly run.

Break down the graph index by relationship type to identify non-lineage edges:

GET <your-namespace>_graph_service_v1_*/_search
{
  "size": 0,
  "aggs": {
    "by_relationship": {
      "terms": {
        "field": "relationshipType",
        "size": 50
      }
    }
  }
}

A healthy graph index should be dominated by DownstreamOf edges. If you see a large volume of IsFailing or IsPassing edges — potentially representing 30–40% or more of the total index — those are metadata test result edges written by the metadata tests job. Each failing test produces one edge per evaluated entity, so a test that evaluates 1 million datasets produces 1 million edges per nightly run.
Estimate daily write churn by examining indexing stats over time. If test-result writes substantially exceed real lineage writes (e.g., 2–3× more), the fragmentation will regenerate after any cleanup until the write source is addressed.

Step 3: Narrow Metadata Test Scope to Reduce Write Churn

This is the prerequisite step before index maintenance. Running a forcemerge without addressing the write source will result in fragmentation returning within the nightly evaluation cycle.

In the DataHub UI, navigate to Govern > Tests (Metadata Tests).
Identify tests that are evaluated against a very broad entity scope — for example, governance scorecard tests (documentation coverage, ownership coverage, description counts) that target your entire data warehouse estate rather than specific domains.
For tests that only report aggregate counts and do not take automated actions, evaluate whether they can be scoped to specific domains or container filters rather than running estate-wide. A test reporting "3% of tables have documentation" across 1 million tables produces the same actionable signal as one scoped per domain, but at a fraction of the edge count.
For action tests (those that assign ownership, set tags, or update properties), apply tighter entity filters so each test runs only against its relevant domain or container, rather than matching the full estate and producing near-duplicate edges for every entity.

Tests can be set to inactive without deletion. This stops the nightly evaluation and stops new edges from being written, and is fully reversible:

# In the DataHub UI: Tests > [select test] > Edit > set status to Inactive
# No entities or test definitions are deleted; the test can be re-activated at any time.

After narrowing or deactivating broad tests, monitor the IsFailing edge count over subsequent nightly cycles. You should see it stop growing and begin declining as the nightly job re-evaluates entities without the inactive tests.

Step 4: Run Forcemerge on the Graph Index (After Write Churn Is Reduced)

Only perform this step after Step 3 is complete. Forcemerging while high write churn is still active adds cluster load without producing lasting benefit.

Schedule the forcemerge during off-peak hours (nights or weekends). This operation is I/O-intensive and will temporarily increase cluster load:
```
POST <your-namespace>_graph_service_v1_<version-suffix>/_forcemerge?max_num_segments=1
```
Verify the operation succeeded by checking that the deleted document count has dropped to near zero and segment count has reduced significantly:
```
GET <your-namespace>_graph_service_v1_*/_stats?filter_path=indices.*.primaries.docs,indices.*.primaries.segments
```
Monitor the deleted document ratio in the days following the merge. If it remains low, the write source has been sufficiently reduced. If it climbs back above ~10% within a week, additional test scope reduction is needed.

Step 5: Address Query Index Retention

Inspect the query index for document age distribution. If the index contains years of SQL query entity history with no TTL or retention policy, implement a retention window appropriate for your use case.
Coordinate with DataHub support or your platform team to implement a retention policy on the query index before running a cleanup, to prevent re-accumulation.

After retention is configured, run a forcemerge on the query index as well:

POST <your-namespace>_queryindex_v2_<version-suffix>/_forcemerge?max_num_segments=1

Step 6: Optional — Enable Search Result Caching During Remediation

As a temporary measure to reduce per-query Elasticsearch pressure while index maintenance is in progress, search result caching can be enabled. This is a configuration change and not a substitute for the structural fixes above.

# Set in your DataHub GMS environment configuration:
SEARCH_SERVICE_ENABLE_CACHE=true

Step 7: Resolve Stale Kubernetes ReplicaSets (If Applicable)

If your environment shows duplicate ReplicaSets for the MAE consumer (e.g., an old set and a new set running simultaneously), clean up the old ReplicaSet. Stale ReplicaSets consume pod IPs and can contribute to AWS CNI IP exhaustion, which blocks auxiliary batch jobs and adds resource pressure.

# Identify duplicate ReplicaSets:
kubectl get replicasets -n <your-namespace> | grep mae-consumer

# Delete the older (lower pod-count) ReplicaSet after confirming the current one is healthy:
kubectl delete replicaset <old-replicaset-name> -n <your-namespace>

Additional Notes

A top-level green Elasticsearch cluster health status does not indicate that individual indices are performing well. Always inspect index-level and node-level metrics (heap utilization, deleted document ratios, segment counts, GC times, write rejection counts) when diagnosing query latency issues. The pattern described in this article — where fragmentation returns shortly after a forcemerge — is a reliable signal that a continuous write source is responsible, and that source must be addressed before index maintenance will produce lasting results. Metadata tests that evaluate against very large entity populations (tens of thousands to millions of entities) are the most common source of this pattern in production deployments. Reducing test scope does not delete any entities or test definitions, and inactive tests can be reactivated at any time. For DataHub Cloud deployments, infrastructure-level operations such as forcemerge, index expansion, and node scaling should be coordinated with DataHub support rather than performed directly.