Preventing DataHub Outages Caused by Overly Complex GraphQL Queries – DataHub Customer Support Portal

Area: API Issues

Sub-Area: GraphQL Query Performance and Complexity

Issue

Excessively complex or wide GraphQL queries sent to the DataHub backend can overwhelm the GMS (Graph Metadata Service), causing cascading failures that manifest as UI unresponsiveness, API timeouts, and HTTP 503 errors. This pattern typically occurs when client applications issue queries with very high field counts, deep nesting, or large page sizes — for example, a searchAcrossEntities query requesting thousands of fields at significant depth, or a scrollAcrossEntities query with a page size of 1,000 and a wide field selection producing multi-megabyte response pages. Because GMS processes these requests synchronously and spends most of its time waiting on Elasticsearch round-trips, response times can reach 18–86 seconds per call. The DataHub API gateway enforces a ~55-second timeout and returns a 503 before GMS completes, and once GMS threads are saturated, interactive UI users are affected as well. This issue can recur if the underlying query patterns are not corrected on the client side.

Error Messages

HTTP 503 Service Unavailable
HTTP 504 Gateway Timeout
GraphQL query timed out
Something went wrong (DataHub UI error page)

You Might Be Asking

Why is the DataHub UI spinning or unresponsive when my GraphQL queries time out?
Why am I getting 503 errors on searchAcrossEntities or scrollAcrossEntities?
Can I cancel a GMS query once it has been sent?
How do I know if my GraphQL query is too complex for DataHub to handle?
Is there a tool to measure GraphQL query complexity before sending it to DataHub?
Why does my DataHub instance recover as soon as I stop running my ingestion pipeline or API consumer?

Solution

The primary fix is on the client side. Reducing query complexity and page size eliminates the root cause; backend guardrails alone cannot fully protect against unusually heavy queries before GMS becomes saturated.

Reduce field selection to only what is needed.

Requesting thousands of fields in a single query is the most common cause of these outages. As a rule of thumb: 500 fields may be acceptable; 1,000 fields is always too many. Audit every query shape your application uses and remove fields that are not consumed by the caller.

Example of an overly broad query pattern to avoid:

# Anti-pattern: requesting all available fields at high depth
query {
  searchAcrossEntities(input: { query: "*", types: [], start: 0, count: 1000 }) {
    searchResults {
      entity {
        urn
        type
        ... on Dataset {
          # dozens of deeply nested sub-selections
          properties { ... }
          ownership { ... }
          tags { ... }
          glossaryTerms { ... }
          schemaMetadata { ... }
          # etc.
        }
      }
    }
  }
}

Preferred pattern — request only what your application actually uses:

# Best practice: select only required fields
query {
  searchAcrossEntities(input: { query: "*", types: ["DATASET"], start: 0, count: 250 }) {
    searchResults {
      entity {
        urn
        type
      }
    }
  }
}

Reduce page size for scroll/search queries.
For scrollAcrossEntities and similar paginated queries, a page size of 1,000 with a wide field selection can produce ~2.5MB response pages that take 18+ seconds to build under no contention. Pages under 1.5MB (achieved at count ≤ 250) consistently complete well under 30 seconds. Start at 250 and increase only if profiling shows it is safe.
```
# Recommended: use a smaller page size for scroll queries
query ScrollEntities($scrollId: String) {
  scrollAcrossEntities(input: {
    types: ["DATASET"],
    query: "*",
    count: 250,        # Keep at or below 250
    scrollId: $scrollId
  }) {
    nextScrollId
    searchResults {
      entity {
        urn
        type
      }
    }
  }
}
```

Measure query complexity before sending.

DataHub's server-side complexity algorithm is open source in the DataHub repository. You can port that logic for an exact match to the server-side thresholds (default limit: 2,000). Alternatively, use a standard client-side library to catch outliers during development or code review:

JavaScript/TypeScript: graphql-query-complexity
Python: graphql-complexity

Example using graphql-query-complexity in JavaScript:

import { getComplexity, simpleEstimator } from 'graphql-query-complexity';
import { parse, buildSchema } from 'graphql';

// Load your DataHub schema
const schema = buildSchema(/* your schema SDL */);

const query = parse(`
  query { searchAcrossEntities(input: { query: "*", count: 250 }) { ... } }
`);

const complexity = getComplexity({
  schema,
  query,
  estimators: [simpleEstimator({ defaultComplexity: 1 })],
});

const COMPLEXITY_LIMIT = 2000;
if (complexity > COMPLEXITY_LIMIT) {
  throw new Error(`Query complexity ${complexity} exceeds limit of ${COMPLEXITY_LIMIT}`);
}
console.log(`Query complexity: ${complexity}`);

Example using graphql-complexity in Python:

from graphql_complexity import get_complexity, SimpleEstimator
from graphql import build_schema, parse

schema = build_schema(open("datahub_schema.graphql").read())

query = parse("""
  query {
    searchAcrossEntities(input: { query: "*", count: 250 }) {
      searchResults { entity { urn type } }
    }
  }
""")

complexity = get_complexity(query, schema, SimpleEstimator(default_complexity=1))

COMPLEXITY_LIMIT = 2000
if complexity > COMPLEXITY_LIMIT:
    raise ValueError(f"Query complexity {complexity} exceeds limit of {COMPLEXITY_LIMIT}")
print(f"Query complexity: {complexity}")

Since most application queries are static templates rather than dynamically generated, the fastest and most effective approach is to review field count and nesting depth for each query shape during code review, before deployment.

Add client-side observability.

Wrap all DataHub API calls with timing and error tracking so you can correlate client-side latency with backend errors and detect regressions early:

# Python example: lightweight request instrumentation
import time
import requests
import logging

def datahub_graphql(query: str, variables: dict = None):
    start = time.time()
    try:
        response = requests.post(
            "https://<your-instance>.datahubproject.io/api/graphql",
            json={"query": query, "variables": variables or {}},
            headers={"Authorization": "Bearer <your-token>"},
            timeout=30,  # Hard client-side timeout
        )
        duration_ms = int((time.time() - start) * 1000)
        logging.info({
            "endpoint": "/api/graphql",
            "duration_ms": duration_ms,
            "status_code": response.status_code,
        })
        response.raise_for_status()
        return response.json()
    except requests.Timeout:
        logging.error("DataHub GraphQL request timed out after 30s — aborting")
        raise

Implement exponential backoff with jitter on 503/504 responses.

Do not retry immediately. Immediate retries amplify load on an already-saturated GMS and worsen the outage.

import time
import random
import requests

def datahub_graphql_with_retry(query: str, variables: dict = None, max_retries: int = 4):
    base_delay = 2.0
    for attempt in range(max_retries):
        response = requests.post(
            "https://<your-instance>.datahubproject.io/api/graphql",
            json={"query": query, "variables": variables or {}},
            headers={"Authorization": "Bearer <your-token>"},
            timeout=30,
        )
        if response.status_code in (503, 504):
            if attempt == max_retries - 1:
                response.raise_for_status()
            delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)
            continue
        response.raise_for_status()
        return response.json()

Avoid concurrent scroll workers and pause automated consumers during incidents.
Running multiple workers that each issue scrollAcrossEntities simultaneously multiplies per-request GMS load. Limit scroll loops to a single concurrent worker. If you detect a spike in 503 errors, pause automated pipelines or batch API consumers immediately — this alone can allow GMS to recover enough for the UI to become usable while the root cause is addressed.

Set a hard client-side request timeout with a circuit breaker.

The server-side GMS timeout is approximately 55 seconds. The client should enforce a shorter timeout (recommended: 30 seconds) and implement a circuit breaker so that if repeated requests fail, the client stops issuing new calls rather than piling up requests against a saturated backend.

# Example: simple circuit breaker state machine (pseudocode)
class CircuitBreaker:
    CLOSED = "closed"      # Normal operation
    OPEN = "open"          # Stop sending requests
    HALF_OPEN = "half_open"  # Test with a single probe

    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.state = self.CLOSED
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.last_failure_time = None

    def record_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = self.OPEN

    def allow_request(self):
        if self.state == self.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = self.HALF_OPEN
                return True
            return False  # Block the request
        return True

Additional Notes

Server-side limits: DataHub enforces several server-side guardrails including a default query complexity threshold of 2,000, query depth limits, a ~55-second async request timeout at the API gateway, and rate limiting. These catch the majority of problematic queries, but an unusually complex query can begin saturating GMS before the complexity limit is evaluated, so client-side prevention is essential.

Cancellation is not possible: Once a GraphQL query has been sent to GMS, there is no mechanism to cancel it. The server will process it to completion or until the ~55-second server-side timeout is reached. This makes pre-flight complexity checks and conservative field selection the only reliable prevention strategy.

Page size guidance: In observed incidents, scrollAcrossEntities with count=1000 and wide hydration produced ~2.5MB pages taking ~18 seconds minimum with no contention. Pages under 1.5MB (achieved at count ≤ 250) consistently completed in under 30 seconds. Start at 250 and profile before increasing.

Impact on interactive users: When GMS threads are fully saturated by heavy background API consumers, interactive UI users are affected as well, because all requests share the same GMS thread pool. Fixing background query patterns protects both programmatic and interactive usage.

Recurrence risk: These issues tend to recur if only the backend load is addressed without fixing client-side query patterns. If a previous incident resolved after infrastructure changes but the same queries continue to run, expect the problem to return, especially as data volume grows.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation