DataHub Instance Slowness Caused by Repeated Full Pulls from Extraction Jobs – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: GMS Performance / Extraction Job Misconfiguration

Issue

A DataHub instance can become severely degraded — exhibiting slow page loads, slow search results, and request timeouts — when an external metadata extraction job repeatedly performs full pulls against the DataHub GMS (Generalized Metadata Service) instead of using incremental (delta) mode. Each full pull issues a large volume of expensive GraphQL queries such as searchLogicalDatasets, searchAcrossEntitiesPhysicalLite, searchAcrossEntitiesURN, and getPhysicalDatasetOverview, returning very large response payloads (tens to hundreds of megabytes per call). These oversized responses consume heap on GMS pods and cause unrelated concurrent queries to stall on the same pod, producing system-wide slowness that is indistinguishable from an infrastructure or product issue until the extraction job is identified as the root cause.

Error Messages

Slow operation entries in GMS logs across multiple GraphQL operations
SocketTimeoutException or ConnectionRequestTimeoutException in GMS logs
SearchTimeoutException during graph or search queries
org.elasticsearch.action.search.SearchPhaseExecutionException indicating shard-level resource pressure

You Might Be Asking

Why is DataHub slow even though infrastructure metrics look normal?
Why are unrelated GraphQL queries timing out at the same time?
Why does my extraction job seem to pull all data every time instead of only new changes?
How do I tell whether a background job is causing GMS slowness?
What does it mean when many GMS responses are byte-for-byte identical and very large?

Solution

Step 1 — Confirm the Extraction Job Is the Root Cause

Collect GMS logs covering a one-hour window during the slowness period. Search for Slow operation log entries and group them by GraphQL operation name.
Look for a small set of operations that account for a disproportionate share of calls and payload volume. A signature pattern is one operation (e.g., searchLogicalDatasets) being called hundreds of times per hour with near-identical or byte-for-byte identical large response sizes, indicating no filtering or pagination is being applied.

Calculate approximate payload throughput across slow operations:

# Pseudo-calculation
total_slow_op_bytes = sum(response_size_bytes for each slow GraphQL call)
# If total_slow_op_bytes >> expected traffic for one hour, an extraction job is likely the cause

Cross-reference the timestamps of slow operations with any scheduled extraction or reconciliation jobs running against the instance. Look for overlap between job execution windows and the slowness window.
Check whether overlapping slow operations occur on the same GMS pod replica. Concurrent large responses on one pod indicate heap pressure and GC-induced stalls affecting all queries on that pod.

Step 2 — Temporarily Pause the Extraction Job to Validate

Pause or disable the extraction job for at least one full hour during a normal load window.
Collect GMS logs for that same window and compare slow-operation counts, payload volumes, and p50/p90/max latencies against the baseline collected in Step 1.
If slow-operation volume drops significantly and overall latency normalizes, the extraction job is confirmed as the primary driver of slowness. No changes to the DataHub deployment itself are required to resolve the issue.

Step 3 — Fix the Extraction Job's Incremental (Delta) Mode

Many extraction and reconciliation jobs are designed to perform a single full pull on first run and then use a timestamp or cursor to fetch only changed records on subsequent runs. If the delta path is not activating, the job falls back to repeated full pulls. Common causes include:

The timestamp or cursor parameter is not being persisted between runs.
The timestamp is being reset to an epoch or default value on each invocation.
A configuration flag enabling incremental mode is missing or set incorrectly.
The job's state store (file, database, or environment variable) is not accessible at startup.

Review the extraction script or recipe to confirm the incremental timestamp parameter is read from a persistent store at startup and written back on successful completion. Example pattern:

# Pseudocode — ensure last_run_timestamp is persisted across invocations
last_run_timestamp = read_from_state_store(key="last_successful_run")

if last_run_timestamp is None:
    # First run: full pull
    records = fetch_all_records()
else:
    # Delta run: only fetch records modified after last run
    records = fetch_records_modified_after(last_run_timestamp)

process(records)
write_to_state_store(key="last_successful_run", value=current_timestamp())

Confirm that the script is passing the timestamp parameter to every DataHub API or GraphQL call that supports time-based filtering. Calls that omit this parameter will return the full dataset regardless of intent.

Add pagination to all list and search queries. Never fetch unbounded result sets in a single call. Use start and count parameters and loop until all pages are retrieved:

# Example paginated GraphQL query pattern
query SearchLogicalDatasets($start: Int!, $count: Int!) {
  searchAcrossEntities(
    input: {
      types: [DATASET],
      query: "*",
      start: $start,
      count: $count
    }
  ) {
    start
    count
    total
    searchResults {
      entity {
        urn
      }
    }
  }
}

After fixing incremental mode, run the extraction job manually in a non-production window and verify that the response payload per run is dramatically smaller than the full-pull baseline.

Step 4 — Schedule Extraction Jobs Outside Peak Hours

Even a correctly implemented incremental extraction job can cause transient load spikes. Schedule extraction jobs during off-peak hours when GMS has headroom to absorb additional query volume.
If the extraction job must run during business hours, introduce rate limiting or request throttling on the extraction side to spread queries over a longer window rather than issuing them in bursts.

Step 5 — Tune GMS for Enterprise-Scale Workloads (On-Premise)

For self-hosted deployments, the default GMS configuration is conservative and may need tuning independently of the extraction job fix. Apply the following environment variable overrides to the GMS deployment (via docker-compose.yml, Kubernetes ConfigMap, or equivalent):

# Elasticsearch connection tuning
ELASTICSEARCH_SOCKET_TIMEOUT=60000          # ms; increase from default 30000
ELASTICSEARCH_CONNECTION_REQUEST_TIMEOUT=10000  # ms; increase from default 5000
ELASTICSEARCH_THREAD_COUNT=8               # increase from default 2 for concurrent load

# Graph traversal tuning
ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS=120  # increase from default 50

# Async request handling
DATAHUB_GMS_ASYNC_REQUEST_TIMEOUT_MS=120000  # ms; increase from default 55000

Additionally, verify Elasticsearch/OpenSearch health:

Disk usage must remain below 75–80%; above this threshold, ES throttles and eventually refuses writes.
Each ES node should have at least 8 GB JVM heap (50% of available RAM, not to exceed 31 GB).
Enable slow query logging on the ES cluster to surface expensive queries independently of GMS logs:

# Enable ES slow query logging (run via ES _cluster/settings API or kibana)
PUT /_cluster/settings
{
  "persistent": {
    "index.search.slowlog.threshold.query.warn": "2s"
  }
}

Additional Notes

The symptom profile described in this article — generalized GMS slowness with large, repeated, byte-for-byte identical GraphQL responses — is distinct from infrastructure-level resource exhaustion. When multiple concurrent slow operations on a single GMS pod replica are explained by a common large-payload query, always investigate external callers (extraction jobs, API clients, polling UI components) before scaling GMS infrastructure. Scaling GMS replicas will distribute load but will not fix a caller that is issuing unbounded full-pull requests; it will simply spread the same excessive load across more pods. The root fix must be on the extraction side. For on-premise deployments, the GMS environment variable tuning in Step 5 is a recommended baseline for any enterprise-scale deployment and is independent of the extraction job issue. If a DataHub upgrade was performed recently and slowness began shortly afterward, review whether new graph traversal logic (multi-hop BFS lineage) is being exercised by lineage-heavy workloads, as this can increase Elasticsearch resource consumption significantly on clusters that were previously operating near capacity.