Avoiding GraphQL Server Overload: Pagination Best Practices for Large Result Sets – DataHub Customer Support Portal

Area: API Issues

Sub-Area: GraphQL Query Performance & Pagination

Issue

Requesting very large result counts (e.g., 10,000 records) in a single GraphQL API call can temporarily overload the DataHub GMS (General Metadata Service) backend, causing widespread instability across the UI, GraphQL API, and SDK. This is because large, unpaginated requests consume significant server memory and processing resources. When such requests are made repeatedly or applied globally — for example, by increasing a "related entities" limit that affects every GMS call — the cumulative load can escalate from intermittent slowness to a complete outage. The DataHub GraphQL API is designed for targeted, paginated queries rather than bulk data export operations.

Error Messages

Query execution timeout
504 Gateway Timeout
Response size too large
Maximum query depth exceeded

You Might Be Asking

Why did increasing my related-entities count cause a full DataHub outage?
What is the maximum safe result count for a single GraphQL query?
How do I paginate through large result sets in the DataHub GraphQL API?
Why is my GraphQL query causing high server load or timeouts?
What are best practices for querying DataHub's GraphQL API at scale?

Solution

Reduce the per-request result count to a safe limit.
Keep the count parameter in any single GraphQL call to approximately 1,000–2,000 results. Requesting 10,000 or more results in a single call is unsupported and will overload the server. If you need more records, iterate over multiple paginated requests.

Use pagination to retrieve large result sets.

Use the start and count parameters to page through results. Increment start by the value of count on each successive call until you have retrieved all records.

# Page 1: records 0–99
query searchDatasetsPage1 {
  search(
    input: {
      type: DATASET
      query: "*"
      start: 0
      count: 100
    }
  ) {
    start
    count
    total
    searchResults {
      entity {
        ... on Dataset {
          urn
          name
        }
      }
    }
  }
}

# Page 2: records 100–199
query searchDatasetsPage2 {
  search(
    input: {
      type: DATASET
      query: "*"
      start: 100
      count: 100
    }
  ) {
    start
    count
    total
    searchResults {
      entity {
        ... on Dataset {
          urn
          name
        }
      }
    }
  }
}

Apply targeted filters to reduce result set size.

Use filters to narrow results by platform, environment, or other attributes. This reduces the data the server must process per query and lowers the total number of pages required.

query searchFilteredDatasets {
  search(
    input: {
      type: DATASET
      query: "sales"
      filters: [
        { field: "platform", values: ["snowflake"] }
        { field: "origin", values: ["PROD"] }
      ]
      start: 0
      count: 100
    }
  ) {
    searchResults {
      entity {
        ... on Dataset {
          urn
          name
        }
      }
    }
  }
}

Request only the fields you actually need.

Selecting many nested aspects (ownership, tags, glossary terms, lineage, etc.) in a single query multiplies the server-side work for every result. Request only the fields required for your use case.

# Avoid: fetching every aspect for every result
query heavyQuery {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,my_table,PROD)") {
    properties { name description }
    ownership { owners { owner { urn } } }
    tags { tags { tag { name } } }
    glossaryTerms { terms { term { name } } }
    editableProperties { description }
    upstreamLineage { upstreams { dataset { urn } } }
  }
}

# Prefer: fetch only what you need
query lightQuery {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,my_table,PROD)") {
    properties {
      name
      description
    }
    platform {
      name
    }
  }
}

Avoid deeply nested lineage queries.

Traversing multiple levels of upstream or downstream lineage in a single query creates exponential load. Query one level at a time and make additional calls as needed.

# Avoid: multiple nested lineage levels in one query
query deepLineage {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,my_table,PROD)") {
    upstreamLineage {
      upstreams {
        dataset {
          upstreamLineage {
            upstreams {
              dataset {
                upstreamLineage {   # Third level — avoid this
                  upstreams { dataset { urn } }
                }
              }
            }
          }
        }
      }
    }
  }
}

# Prefer: one level at a time
query shallowLineage {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,my_table,PROD)") {
    upstreamLineage {
      upstreams {
        dataset {
          urn
          properties { name }
        }
      }
    }
  }
}

Be aware of global impact when changing entity-count limits.
Configuration changes that increase the number of related entities returned (such as a "related entities" limit) apply to every GMS call made by your application — not just the specific query you intend to expand. A limit of 10,000 on a globally applied setting can cause every single GMS request to attempt to fetch that many related entities simultaneously, rapidly exhausting server resources. Evaluate the blast radius of any limit increase before applying it to production.
Use the REST API for bulk export operations.
If you need to export large volumes of metadata, consider using the DataHub REST API rather than GraphQL. The GraphQL endpoint is optimized for UI interactions and targeted queries. For bulk operations, the REST API provides more appropriate endpoints.

Additional Notes

The DataHub GraphQL API does not support result counts of 10,000 per call and such requests may temporarily destabilize the GMS service, affecting all users of the instance — not just the caller. A per-call limit of 1,000–2,000 results is the recommended maximum for standard use. If you rolled back a large-count configuration and still observe timeouts, check whether any other query paths retained the higher limit, and share the active queries with DataHub Support for further diagnosis. For DataHub Cloud deployments, server-side resource management is handled by the DataHub Cloud operations team; customers should open a support ticket if instability persists after reducing query sizes.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation