Reducing GraphQL Payload Size and UI Slowness Caused by Sibling Asset Fetching – DataHub Customer Support Portal

```html

Area: UI Issues

Sub-Area: GraphQL Performance / Sibling Asset Overfetching

Issue

In DataHub deployments where many assets have sibling relationships (for example, a dbt model linked to a Snowflake table), the UI can become noticeably slow on search result pages, data product pages, and asset profile pages. The root cause is that DataHub's client-side sibling merge logic requires fetching the full detail set for every sibling of every asset rendered on the page — including metadata fields such as institutional memory, dataset profiles, usage statistics, and lineage relationships — even when only minimal information (asset name, platform URN, etc.) is actually needed for display. In deployments with a large number of sibling-bearing assets, this results in very large GraphQL response payloads, increased load on GMS and OpenSearch/Elasticsearch, slow page load times, and in severe cases, HTTP 500 errors from the GraphQL layer.

Error Messages

ERROR c.datahub.graphql.GraphQLController - Errors while executing query:
HTTP 500 Internal Server Error on GraphQL endpoint

You Might Be Asking

Why does the DataHub UI load slowly when assets have sibling relationships?
Why are GraphQL responses so large on search and data product pages?
How can I reduce the number of fields fetched for sibling assets in search results?
What environment variables can I set to improve DataHub UI performance?
Why does enabling sibling/lineage features cause GMS and OpenSearch to become overloaded?

Solution

There are two complementary approaches to address this problem: enabling environment variable flags that suppress unnecessary sibling and lineage fetching, and upgrading to a release that includes GraphQL query optimizations.

Enable the SHOW_SEPARATE_SIBLINGS environment variable (v0.3.17+)
When SHOW_SEPARATE_SIBLINGS is set to true, DataHub stops merging sibling assets on the client side and instead displays each sibling as a separate entity. This eliminates the need to fetch full sibling detail sets on every page render, significantly reducing payload size and GMS/OpenSearch load.

Set this variable in your datahub-frontend deployment configuration:
```
# Helm values.yaml (datahub-frontend extraEnvs) or equivalent
- name: SHOW_SEPARATE_SIBLINGS
  value: "true"
```
Or as a Docker environment variable:
```
SHOW_SEPARATE_SIBLINGS=true
```
Enable the HIDE_LINEAGE_IN_SEARCH_CARDS environment variable
Fetching lineage information for every asset displayed in a search results page adds significant overhead, especially when lineage ingestion is enabled. Setting HIDE_LINEAGE_IN_SEARCH_CARDS to true suppresses lineage badge queries on search cards, removing a source of high-cost graph traversal queries per search page load.
```
# Helm values.yaml (datahub-frontend extraEnvs) or equivalent
- name: HIDE_LINEAGE_IN_SEARCH_CARDS
  value: "true"
```
Or as a Docker environment variable:
```
HIDE_LINEAGE_IN_SEARCH_CARDS=true
```

Apply both flags together in Helm

A combined example for a Helm-based deployment:

datahub-frontend:
  extraEnvs:
    - name: SHOW_SEPARATE_SIBLINGS
      value: "true"
    - name: HIDE_LINEAGE_IN_SEARCH_CARDS
      value: "true"

After applying the changes, restart the datahub-frontend pods:

kubectl rollout restart deployment/datahub-frontend -n <your-namespace>

Upgrade to a release with sibling sub-fetch optimizations
DataHub engineering has shipped targeted GraphQL fragment optimizations that remove heavyweight fields (institutionalMemory, datasetProfiles(limit:1), statsSummary.topUsersLast30Days) from the sibling sub-fetch used in search and data product queries. These fields are not consumed by the client-side sibling merge logic but were previously fetched for every sibling on every page. The optimization results in approximately 20–25% smaller response payloads and faster response times for pages with sibling-bearing assets. Confirm with your DataHub account team which release version includes these changes for your deployment track.
Verify improvement after applying changes
After enabling the flags and/or upgrading, use your browser's developer tools (Network tab) to inspect the GraphQL response payload on a search results page or data product page that previously showed slowness. Compare the response size and time before and after the change. You can also monitor GMS logs and OpenSearch metrics for a reduction in query load.
```
# Example: watch GMS logs for GraphQL errors after applying the fix
kubectl logs -f deployment/datahub-gms -n <your-namespace> | grep -i "graphql\|error\|slow"
```
Address OpenSearch connection pool exhaustion (if HTTP 500 errors persist)
In deployments where the sibling overfetching has already caused significant load, you may also see OpenSearch/Elasticsearch connection pool exhaustion. If GraphQL 500 errors continue after applying the environment variable flags, review the following GMS configuration settings and increase them as needed:
- OpenSearch HTTP connection pool size
- Maximum connections per route
- Connection timeout and lease timeout values
Check full GMS logs (not just truncated snippets) for connection lease timeout exceptions to confirm whether pool exhaustion is contributing to failures.

Additional Notes

The SHOW_SEPARATE_SIBLINGS and HIDE_LINEAGE_IN_SEARCH_CARDS environment variables are available starting in DataHub version v0.3.17 and later. If you are running an older version, the primary mitigation is to upgrade. The client-side sibling merge design — which necessitates fetching full sibling detail sets — is a known architectural limitation that the DataHub engineering team is actively working to eliminate in future releases; the environment variable flags and GraphQL fragment optimizations described above are interim mitigations. Enabling SHOW_SEPARATE_SIBLINGS changes how sibling assets are displayed in the UI (each sibling appears as a distinct row rather than being merged into one card), so evaluate whether this presentation is acceptable for your users before enabling in production. The lineage count suppression flag (HIDE_LINEAGE_IN_SEARCH_CARDS) is particularly impactful in deployments where lineage ingestion is enabled across many assets, as without it each search page can generate dozens of graph traversal queries against OpenSearch.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation