GMS GraphQL Timeouts After Version Upgrade: Aligning Client and Server Timeout Settings – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: GMS Performance / GraphQL Timeout Misconfiguration

Issue

After upgrading DataHub, pipeline jobs that issue GraphQL requests to the GMS endpoint — particularly scrollAcrossLineage, scrollAcrossEntities, and domain/parent-domain lookups — begin failing with repeated read timeouts. The client's HTTP read timeout (commonly 30 seconds) is shorter than the time GMS actually needs to complete lineage graph traversal queries against Elasticsearch or OpenSearch. Because the client is configured to retry on failure, each failing GraphQL call consumes roughly timeout × retry_count seconds before giving up, causing throughput to collapse dramatically when processing large numbers of entities. The issue is not a client logic error; GMS is genuinely slower after the upgrade, and the mismatch between the client-side read timeout and the GMS-side graph query timeout turns every slow response into a hard failure and a costly retry storm.

Error Messages

ReadTimeoutError: HTTPSConnectionPool(host='<your-gms-host>', port=443): Read timed out. (read timeout=30.0)
Max retries exceeded with url: /api/gms/api/graphql
Issue fetching DOWNSTREAM lineage for <urn>

You Might Be Asking

Why did my pipeline start timing out on GraphQL calls only after a DataHub version upgrade?
Why does each failed lineage query take 2+ minutes instead of just 30 seconds?
Is the timeout caused by a bug in my client code or by GMS being slow?
What is ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS and how does it relate to my client timeout?
How do I stop the retry storm while I investigate the root cause?

Solution

Address this in two phases: an immediate stopgap to halt the retry storm, and a root-cause investigation to restore GMS query performance.

Phase 1 — Immediate Stopgap: Align Client Timeout with GMS Graph Timeout

Determine the GMS-side graph query timeout. Check the GMS environment configuration for the following setting:
```
ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS
```
If this variable is not explicitly set, the default is typically 50–60 seconds depending on your DataHub version. You can also check the equivalent Spring property:
```
search.graph.timeoutSeconds
```
Set your HTTP client's read timeout to a value greater than the GMS graph timeout. A buffer of 10–30 seconds above the GMS timeout is recommended. For example, if the GMS graph timeout is 50 seconds, set the client read timeout to at least 60–80 seconds.

Example — Python requests-based client:
```
import requests

response = requests.post(
    "https://<your-gms-host>/api/gms/api/graphql",
    json={"query": "<your-graphql-query>"},
    timeout=80  # Must exceed ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS
)
```
Example — Airflow HTTP operator or custom hook:
```
# In your Airflow connection or hook configuration
# Set the read timeout on the underlying session:
session.timeout = 80  # seconds
```
If you control the GMS configuration and the graph timeout is higher than needed, you can instead lower it so GMS returns an explicit error before the client times out. Update the GMS deployment environment:
```
# In your GMS deployment (e.g., Helm values or Docker environment):
ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS=45
```
Then set your client timeout to at least 55 seconds. The key invariant is:
```
client_read_timeout > ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS
```

Phase 2 — Root Cause Investigation: Restore GMS Query Performance

Check for a known v1.4.x performance regression (OSS issue fixed in v1.5.0+). A confirmed regression in DataHub v1.4.x introduced an uncached synchronous Elasticsearch indexExists() call inside ElasticSearchService.appendRunId(), firing on every metadata change event. Under moderate ingestion load this saturates the MCL consumer thread pool and causes general GMS unresponsiveness including slow GraphQL query responses. Inspect GMS logs for high-frequency patterns such as:
```
grep -i "indexExists\|appendRunId\|RejectedExecutionException\|Thread pool is full" <gms-log-file>
```
If these messages appear at high frequency, upgrade GMS to v1.5.0 or later, which contains the fix. Version v1.6.0 additionally sets ES_BULK_REFRESH_POLICY=NONE by default, further reducing Elasticsearch write pressure.

Check Elasticsearch / OpenSearch cluster health. Lineage queries (scrollAcrossLineage) execute ES queries per hop for graphs under 300 hops. A degraded ES cluster directly causes GMS query stalls.

# Check cluster health
curl -X GET "https://<your-elasticsearch-host>:9200/_cluster/health?pretty"

# Check node JVM heap usage
curl -X GET "https://<your-elasticsearch-host>:9200/_nodes/stats/jvm?pretty" \
  | grep -A5 "heap_used_percent"

# Check for slow queries (requires slow log enabled)
curl -X GET "https://<your-elasticsearch-host>:9200/graph_service_v1*/_settings?pretty" \
  | grep -i "slowlog"

If JVM heap usage exceeds 70–80%, add nodes or increase heap allocation. Relevant index patterns: graph_service_v1* and entityv2*.

Check GMS pod resource utilization and logs for thread exhaustion.

# Kubernetes example — check resource usage
kubectl top pods -n <your-datahub-namespace> -l app=datahub-gms

# Stream GMS logs and filter for errors during a failure window
kubectl logs -n <your-datahub-namespace> deployment/datahub-gms --since=10m \
  | grep -iE "RejectedExecution|Thread pool|WARN|ERROR|timeout"

Increase GMS JVM heap if under-provisioned. For large entity graphs, the default heap of 1–2 GB is often insufficient. Set the following in your GMS deployment configuration:
```
# Helm values.yaml example
datahub-gms:
  extraEnvs:
    - name: JAVA_OPTS
      value: "-Xms2g -Xmx4g"
```
Verify ES_BULK_REFRESH_POLICY is set to NONE. On v1.5.0+ this should be the default, but confirm it is not set to WAIT_UNTIL, which significantly increases ES write latency.
```
# In GMS environment configuration
ES_BULK_REFRESH_POLICY=NONE
```
Account for lineage cache cold-start after GMS restarts. After any GMS restart (including those triggered by an upgrade), the in-memory lineage search cache starts completely cold (default TTL: 24 hours). Every unique scrollAcrossLineage query hits Elasticsearch from scratch until the cache warms. Expect elevated latency for a period after any GMS restart, particularly for large graphs. To bypass the cache for diagnostic purposes, use:
```
{
  "query": "query { scrollAcrossLineage(...) }",
  "variables": {
    "searchFlags": { "skipCache": true }
  }
}
```

Reduce query complexity during degraded periods. Limit lineage hops, use pagination with smaller page sizes, and avoid wildcard queries until GMS performance is restored:

# Example: limit to 1 hop and a small count
query ScrollLineage {
  scrollAcrossLineage(
    urn: "<your-entity-urn>"
    direction: DOWNSTREAM
    count: 10
    maxHops: 1
  ) {
    searchResults { entity { urn } }
  }
}

Additional Notes

The root timeout mismatch can occur even when GMS is functioning correctly if the graph walk for a given entity legitimately takes longer than the client's read timeout. Always verify ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS before concluding that GMS is broken.
The performance regression described in Phase 2, Step 1 was introduced in DataHub v1.4.x and fixed in v1.5.0. If you upgraded to a version within the v1.4.x range, upgrading further to v1.5.0+ is the recommended resolution.
In self-hosted (on-premise or self-managed Kubernetes) deployments, all infrastructure scaling — GMS, Elasticsearch, Kafka, and networking — is the operator's responsibility. There is no managed autoscaling layer as in DataHub Cloud.
A separate known issue (backlog) exists where searchAcrossLineage may cache empty results during an Elasticsearch index build, causing lineage to appear missing until GMS restarts. This manifests as silent empty results rather than timeouts. Use searchFlags: { skipCache: true } as a diagnostic workaround.
The retry multiplier effect (timeout × retry_count) is a significant throughput amplifier. With a 30-second timeout and 4 attempts, each failing entity costs ~2 minutes. Raising the timeout without fixing the root cause reduces retries but does not restore throughput; both must be addressed.