Optimizing GraphQL Queries to Avoid API Timeouts When Fetching Glossary Term Related Assets – DataHub Customer Support Portal

Area: API Issues

Sub-Area: GraphQL Query Performance & Optimization

Issue

Pipelines that programmatically fetch all assets related to a glossary term using the relationships GraphQL field can experience significant API timeouts and slow response times. When a glossary term has a large number of tagged assets (100+), the getGlossaryTermRelatedAssets pattern — which retrieves all incoming TermedWith relationships and then filters client-side — places excessive load on the Elasticsearch layer, causing connection lease timeouts and repeated read timeouts on the /gms/api/graphql endpoint. This is especially problematic when the query is executed in a loop for many glossary terms in a single pipeline run.

Error Messages

com.datahub.util.exception.ESQueryException: Search query failed:. Root cause: Search query failed:. error while performing request. Connection lease request time out
ReadTimeoutError("HTTPSConnectionPool(host='<your-instance>.acryl.io', port=443): Read timed out. (read timeout=30.0)")
datahub.configuration.common.GraphError: Error executing graphql query: [{'message': 'com.datahub.util.exception.ESQueryException: ...', 'extensions': {'code': 500, 'type': 'SERVER_ERROR', 'classification': 'DataFetchingException'}}]

You Might Be Asking

Why does my pipeline time out when fetching assets tagged with a specific glossary term?
How can I efficiently query only the assets related to a glossary term that also match a platform or tag filter?
Is there a better alternative to using the relationships field on a glossaryTerm query when I need to apply additional filters?
Why does my GraphQL query return too much data and cause Elasticsearch connection lease timeouts?

Solution

Identify the inefficient query pattern. The following query fetches all related assets for a glossary term without any server-side filtering, returning every relationship and requiring the client to filter the results afterward. When a term has many associated assets, this query scans a large dataset in Elasticsearch and frequently times out:
```
query getGlossaryTermRelatedAssets($urn: String!) {
  glossaryTerm(urn: $urn) {
    relationships(input: {
      types: ["TermedWith"]
      direction: INCOMING
    }) {
      relationships {
        entity {
          urn
          type
        }
      }
    }
  }
}
```

Replace with a searchAcrossEntities query that applies filters server-side. Instead of fetching all related assets and filtering client-side, push the filter conditions directly into the GraphQL query using searchAcrossEntities with orFilters. The example below retrieves only assets that are tagged with a specific glossary term AND either belong to a specific platform OR carry a specific tag — all in a single efficient server-side query:

query GetHighPrioRelatedAssets($termUrn: String!) {
  searchAcrossEntities(
    input: {
      query: "*"
      count: 50
      orFilters: [
        {
          and: [
            { field: "glossaryTerms", values: [$termUrn] }
            { field: "platform", values: ["urn:li:dataPlatform:<your-platform>"] }
          ]
        }
        {
          and: [
            { field: "glossaryTerms", values: [$termUrn] }
            { field: "tags", values: ["urn:li:tag:<your-tag-id>"] }
          ]
        }
      ]
    }
  ) {
    searchResults {
      entity {
        urn
      }
    }
  }
}

With example variables:

{
  "termUrn": "urn:li:glossaryTerm:<your-glossary-term-name>"
}

Paginate large result sets. If the number of matching assets may exceed your count value, implement pagination using start and count parameters to avoid fetching an unbounded result set in a single request:

query GetHighPrioRelatedAssetsPaginated($termUrn: String!, $start: Int!, $count: Int!) {
  searchAcrossEntities(
    input: {
      query: "*"
      start: $start
      count: $count
      orFilters: [
        {
          and: [
            { field: "glossaryTerms", values: [$termUrn] }
            { field: "platform", values: ["urn:li:dataPlatform:<your-platform>"] }
          ]
        }
      ]
    }
  ) {
    total
    searchResults {
      entity {
        urn
      }
    }
  }
}

Add retry logic with exponential backoff. Even with an optimized query, transient timeouts can occur during periods of elevated platform load. Implement retry logic in your pipeline to handle these gracefully:

import time
import datahub.ingestion.graph.client as datahub_client
from datahub.configuration.common import GraphError

def execute_with_retry(graph_client, query, variables, max_retries=3, base_delay=5):
    for attempt in range(max_retries):
        try:
            return graph_client.execute_graphql(query, variables)
        except GraphError as e:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt)
            print(f"GraphQL error on attempt {attempt + 1}, retrying in {delay}s: {e}")
            time.sleep(delay)

Reduce pipeline concurrency during high-load periods. If your pipeline executes queries for many glossary terms in parallel, temporarily reducing the number of concurrent workers lowers pressure on the Elasticsearch layer and reduces the likelihood of connection lease timeouts.

Additional Notes

The relationships field on entity queries (e.g., glossaryTerm { relationships(...) }) is a general-purpose traversal API that is not optimized for large fan-out scenarios. When a glossary term is applied to hundreds or thousands of assets, this field must resolve a very large number of Elasticsearch documents, which can exhaust connection pool resources and cause cascading timeouts. The searchAcrossEntities API is purpose-built for filtered asset discovery and is significantly more efficient for use cases that require combining entity type, platform, tag, or glossary term filters. Always prefer server-side filtering over client-side filtering when working with large datasets. The count parameter in searchAcrossEntities defaults to a platform-defined maximum; consult your DataHub instance documentation for the applicable limit and paginate accordingly.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation