UI Slowness and 503 Errors Caused by Large scrollAcrossEntities GraphQL Queries – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: GraphQL Performance / Elasticsearch Query Optimization

Issue

Users and background services may experience intermittent HTTP 503 errors and general UI slowness when DataHub's scrollAcrossEntities GraphQL endpoint is called with large page sizes and wide field selections. The API gateway enforces a hard timeout (typically around 55 seconds); when GMS takes longer than that to fulfill a scroll request — due to the volume of data being hydrated per page, combined with Elasticsearch index fragmentation or cluster heap pressure — the gateway terminates the connection and returns a 503 before a response is delivered. Interactive users see slow page loads on lineage and Discover views, while background services issuing programmatic scroll loops receive elevated 5xx error rates. In severe cases, the same Elasticsearch index that serves lineage queries accumulates tens of millions of additional edges from estate-wide metadata test results, compounding write churn and keeping fragmentation persistently high even after manual cleanup operations.

Error Messages

HTTP 503 Service Unavailable on POST /api/graphql
HTTP 504 Gateway Timeout on /gms/aspects
scrollAcrossEntities took [12000–86000] ms (in GMS logs)

You Might Be Asking

Why does my scrollAcrossEntities loop return 503 errors intermittently but not always?
Why is lineage slow to load in the DataHub UI even though Elasticsearch reports a green cluster status?
Will reducing the number of ingested entities fix UI slowness?
Why did forcemerging the graph index not permanently fix lineage query latency?
How does metadata test configuration affect Elasticsearch performance?

Solution

1. Reduce Page Size on scrollAcrossEntities Calls

The per-request cost of a scrollAcrossEntities query scales roughly linearly with the count parameter. At count=1000, a single page with a wide field selection can take 18 seconds or more under no contention, and longer under any shared load. Reducing page size is the single most effective immediate mitigation.

Set count to 250 or lower in all scrollAcrossEntities calls. Pages at or below approximately 1.5 MB consistently complete well under 30 seconds in observed production workloads.
Trim the fields requested per entity. Request only the fields your application actually uses. Example of a minimal field selection:

query ScrollEntities($input: ScrollAcrossEntitiesInput!) {
  scrollAcrossEntities(input: $input) {
    nextScrollId
    searchResults {
      entity {
        urn
        type
      }
    }
  }
}

Avoid running multiple concurrent scroll loops against the same endpoint from separate workers. Concurrent executions multiply per-request backend load and dramatically increase the probability of crossing the gateway timeout threshold.
Apply exponential backoff with jitter on all 503 responses. Do not retry immediately.

import time, random

def scroll_with_backoff(client, input_vars, max_retries=5):
    delay = 1.0
    for attempt in range(max_retries):
        response = client.execute(SCROLL_QUERY, variables=input_vars)
        if response.status_code == 503:
            sleep_time = delay + random.uniform(0, delay * 0.5)
            time.sleep(sleep_time)
            delay = min(delay * 2, 60)
            continue
        response.raise_for_status()
        return response.json()
    raise RuntimeError("Exceeded max retries on scrollAcrossEntities")

2. Diagnose Elasticsearch Index Fragmentation

A green cluster-level Elasticsearch status does not rule out index-level fragmentation or node-level heap pressure. Both conditions can cause slow PIT (point-in-time) queries that underlie lineage traversal and Discover page loads without surfacing in top-level health checks.

Check deleted document ratios on key indices (values above ~10–15% degrade query performance):

GET /_cat/indices/_graph_service_v1*?v&h=index,docs.count,docs.deleted,store.size&s=store.size:desc

Check node-level heap usage and write rejection counts:

GET /_cat/nodes?v&h=name,heap.percent,cpu,node.role,search.query_total,indexing.index_failed&s=heap.percent:desc

Check PIT query cumulative latency on the graph index:

GET /_graph_service_v1*/_stats/search?filter_path=indices.*.total.search.point_in_time*

If the deleted document ratio on graph_service_v1 exceeds 15%, run a forcemerge during off-peak hours to reclaim deleted segments and restore PIT query performance:
```
POST /_graph_service_v1_*/_forcemerge?max_num_segments=1
```
Important: Do not run forcemerge while the write source causing fragmentation is still active. See Step 3 below.

3. Identify and Reduce Estate-Wide Metadata Test Write Churn

A frequently overlooked cause of persistent graph_service_v1 fragmentation is estate-wide metadata tests (assertions). Each failing test produces one graph edge per evaluated entity. If a test evaluates against millions of datasets and fails on most of them, it generates millions of IsFailing edges into the same index that serves lineage queries, and regenerates them on every nightly job run. This pattern causes fragmentation to return within days of a forcemerge.

Inspect the relationship type breakdown of your graph index to determine what fraction of edges are test results vs. real lineage:
```
GET /_graph_service_v1*/_search
{
  "size": 0,
  "aggs": {
    "by_relationship": {
      "terms": { "field": "relationshipType", "size": 50 }
    }
  }
}
```
A high proportion of IsFailing or IsPassing edges relative to DownstreamOf edges indicates metadata tests are a primary write source.
In the DataHub UI, navigate to Govern > Tests (or Observe > Assertions depending on your version) and identify tests that are scoped to your full entity estate (e.g., all datasets across a large platform like Snowflake or BigQuery) and are failing at high rates.
For governance scorecard tests (e.g., documentation coverage, ownership coverage, description counts) that evaluate against the full estate, consider either:
- Narrowing the test filter to specific domains, containers, or tags rather than running estate-wide, or
- Setting the test to Inactive while re-assessing its scope. Setting a test to inactive removes it from the nightly evaluation entirely and stops new result edges from being written. This is fully reversible.
For action-type tests (e.g., auto-assign ownership, propagate deprecated status), narrow the selection filter to the relevant domain or container rather than disabling the test, so the action continues to apply where intended.
Once the nightly test job has run at least once after scope reduction, confirm that the IsFailing edge count is declining, then run the forcemerge from Step 2. The cleanup will hold this time because the write source is reduced.

4. Additional Elasticsearch Stability Settings

If your deployment supports it, enable the DataHub search result cache to reduce per-query Elasticsearch pressure during high-load periods:
```
# In GMS environment configuration
SEARCH_SERVICE_ENABLE_CACHE=true
```
If lineage traversal depth is configured at a high value (e.g., 20 hops), consider reducing it for deployments where deep lineage is not required, as each additional hop adds Elasticsearch round-trips:
```
# In GMS environment configuration
ELASTICSEARCH_SEARCH_GRAPH_LINEAGE_MAX_HOPS=10
```

Additional Notes

A general entity catalog cleanup (deleting ingested datasets or other entities broadly) will not resolve performance issues caused by Elasticsearch index fragmentation or oversized metadata test result indices. Targeted remediation — reducing scroll page sizes, forcemerging fragmented indices, and narrowing metadata test scope — is required. The API gateway timeout (approximately 55 seconds in DataHub Cloud) is a fixed ceiling; the only durable fix for 503s on scroll queries is ensuring each individual request completes within that window, which is controlled by page size and field selection on the client side. Cluster-level Elasticsearch health (green status, acceptable average search latency) does not rule out index-level or node-level conditions that degrade specific query patterns. Always check per-index deleted document ratios and per-node heap and GC metrics when investigating intermittent GraphQL timeouts. For DataHub Cloud deployments, infrastructure-level remediation (forcemerge execution, cluster expansion, shard rebalancing) must be performed by the DataHub platform team — contact DataHub Support to request these operations.