Handling Intermittent 503 Errors on GraphQL scrollAcrossEntities API Calls – DataHub Customer Support Portal

Area: API Issues

Sub-Area: GraphQL Pagination / scrollAcrossEntities

Issue

When building integrations that use the DataHub GraphQL scrollAcrossEntities API to paginate through large result sets, clients may intermittently receive HTTP 503 errors. These errors are transient and are not caused by a malformed query or invalid credentials. They typically occur because the API gateway closes the connection before the backend finishes resolving the request under elevated load conditions — particularly when the underlying search and entity-resolution services experience resource contention. The 503 is returned well before a typical client-side timeout fires, so increasing the timeout alone does not prevent these errors.

Error Messages

503 Service Unavailable
httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'https://<your-instance>.acryl.io/api/graphql'

You Might Be Asking

Why am I getting 503 errors when calling scrollAcrossEntities even though my query is correct?
Will increasing my HTTP client timeout fix the 503 errors on the DataHub GraphQL endpoint?
Is there something wrong with my pagination loop or my scrollId handling?
How should I handle transient 503 errors when iterating through paginated GraphQL results?

Solution

The query pattern itself and standard scroll-loop logic are correct. The 503s are transient backend timeouts. Apply the following client-side mitigations to make your integration resilient:

Add exponential backoff with retry logic. Wrap each GraphQL request in a retry loop that catches 503 responses and waits before retrying. Do not call raise_for_status() before checking whether the status code is retryable.

import asyncio
import httpx

GRAPHQL_URL = "https://<your-instance>.acryl.io/api/graphql"
AUTH_TOKEN  = "<your-datahub-token>"

SCROLL_QUERY = """
query ScrollAcrossEntities($scrollId: String, $count: Int) {
    scrollAcrossEntities(input: {
        types: [DATA_PRODUCT],
        count: $count,
        query: "*",
        scrollId: $scrollId
    }) {
        nextScrollId
        count
        total
        searchResults {
            entity {
                urn
                type
            }
        }
    }
}
"""

async def graphql_post_with_retry(client, payload, max_retries=5):
    delay = 2  # seconds
    for attempt in range(max_retries):
        resp = await client.post(
            GRAPHQL_URL,
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {AUTH_TOKEN}",
            },
            json=payload,
        )
        if resp.status_code == 503:
            if attempt < max_retries - 1:
                await asyncio.sleep(delay)
                delay = min(delay * 2, 60)  # cap at 60 seconds
                continue
            resp.raise_for_status()
        else:
            resp.raise_for_status()
            return resp
    raise RuntimeError("Exceeded maximum retries for GraphQL request")


async def scroll_all_entities(page_size=500):
    async with httpx.AsyncClient(timeout=120) as client:
        scroll_id = None
        while True:
            payload = {
                "query": SCROLL_QUERY,
                "variables": {"scrollId": scroll_id, "count": page_size},
            }
            resp = await graphql_post_with_retry(client, payload)
            data = resp.json()["data"]["scrollAcrossEntities"]
            for result in data["searchResults"]:
                yield result["entity"]
            scroll_id = data.get("nextScrollId")
            if not scroll_id:
                break

Reduce the page size per request. Smaller pages reduce the per-request work performed by the backend search and entity-resolution layers. A page size of 500–1000 entities is recommended as a starting point. Avoid requesting the maximum possible count in a single call when the total data set is large.
Avoid running multiple heavy paginated queries concurrently. If your service spawns several parallel scroll loops against the same endpoint, the aggregate load significantly increases the probability of a 503. Serialize or throttle concurrent scroll operations.
Do not rely solely on raising the client timeout. Because the API gateway returns the 503 before the backend query completes, a higher timeout value on the client side does not prevent the gateway from closing the connection early. Retry logic is the correct mitigation.
Contact DataHub Support if 503s are frequent or sustained. Transient 503s under load are expected to be rare. If errors become frequent or persist over multiple hours, open a support ticket with representative timestamps and the affected operation names so the platform team can review backend metrics and apply infrastructure-level mitigations.

Additional Notes

In DataHub Cloud (Acryl-managed), all backend infrastructure — including the API gateway, Elasticsearch cluster, GMS service, and underlying database — is fully managed by the DataHub platform team. Customers do not have direct access to scale or tune these components. Transient 503s caused by backend resource contention are resolved through platform-level changes deployed by the engineering team. Client-side retry logic with exponential backoff is the recommended and supported pattern for building resilient integrations against any cloud-hosted GraphQL endpoint and is documented in the DataHub GraphQL best practices guide. The same retry pattern is applicable to both scrollAcrossEntities and searchAcrossEntities queries, as both involve the same entity-resolution step that is susceptible to gateway timeouts under elevated load.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation