Ingestion Jobs Stuck In Progress Due to GMS Query String Length Limit – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Executor Coordinator / Sweeper Job Failures

Issue

When running multiple ingestion sources in parallel (for example, five concurrent recipes ingesting a large number of datasets), ingestion jobs may appear stuck in an In Progress state in the DataHub UI for hours even after all data has been successfully ingested. Cancel attempts also fail to transition the job status. The underlying cause is that the executor coordinator's sweeper job — which is responsible for transitioning ingestion run statuses from RUNNING to SUCCESS or ABORTED — fails on every cycle with a 400 Bad Request error when it queries GMS for recent execution requests. This happens because the Elasticsearch query string constructed from multiple ingestion source URNs exceeds the default SEARCH_VALIDATION_MAX_QUERY_LENGTH limit enforced by GMS (default: 500 characters). With five or more ingestion sources, the combined query string grows to approximately 150–200 characters per source, easily exceeding this threshold. A secondary contributing factor seen in high-parallelism scenarios is the Kafka consumer's max.poll.interval.ms being too short, causing the executor coordinator to be evicted from its Kafka consumer group and report health check failures (HTTP 500).

Error Messages

Sweeper error: 400 Client Error: Bad Request for url: http://<gms-host>:8080/openapi/v2/entity/dataHubExecutionRequest?query=((ingestionSource:"urn:li:dataHubIngestionSource:<SOURCE_ID>" AND requestTimeMs: >=...) OR (...)) AND ...
Application maximum poll interval (10000ms) exceeded by Xms (adjust max.poll.interval.ms for long-running message processing): leaving group
Sweeper(XXXXXXXXXX): will skip RUNNING ingestion <run-id> in source urn:li:dataHubIngestionSource:<SOURCE_ID>.
Sweeper(XXXXXXXXXX): plan execution finished: 0 of 0 actions succeeded.
Failed to load RemoteExecutorStatus record with urn urn:li:dataHubRemoteExecutor:<executor-id>: com.linkedin.pegasus2avro.executor.RemoteExecutorStatus is missing required field: executorPoolId

You Might Be Asking

Why does my ingestion show "In Progress" for hours even though all datasets have been successfully ingested?
Why does canceling an in-progress ingestion job have no effect?
Why does the sweeper log show "0 of 0 actions succeeded" on every cycle?
Why does parallel ingestion work fine in one environment but get stuck in another environment running the same version?
Why is my executor coordinator reporting HTTP 500 health check failures during large ingestion runs?

Solution

Apply the following two configuration changes to your Helm deployment. Both address distinct but related failure modes that together cause ingestion jobs to become permanently stuck in the In Progress state.

Increase the GMS search query length limit.
The sweeper queries GMS for all recent execution requests across all active ingestion sources in a single URL-encoded query string. With five or more sources, this string exceeds GMS's default 500-character validation limit, causing a 400 error and preventing the sweeper from processing any status transitions. Raise the limit to accommodate your number of parallel ingestion sources.

Add the following environment variable to your datahub-gms deployment via values.yaml:
```
datahub-gms:
  extraEnvs:
    - name: SEARCH_VALIDATION_MAX_QUERY_LENGTH
      value: "2000"
```
Tune the value as needed. A value of 2000 is sufficient for up to approximately ten concurrent ingestion sources. Increase further if you have more sources.
Increase the Kafka max poll interval for the executor coordinator.
Under high parallelism, the executor coordinator's Kafka consumer can take longer than the default max.poll.interval.ms (10,000 ms) to complete a processing loop, causing it to be forcibly evicted from the consumer group. This results in repeated HTTP 500 health check failures and prevents the coordinator from processing status update messages. Raise the interval to 5 minutes (300,000 ms).

Add the following environment variable to your acryl-datahub-executor-coordinator deployment via values.yaml:
```
acryl-datahub-executor-coordinator:
  extraEnvs:
    - name: KAFKA_PROPERTIES_MAX_POLL_INTERVAL_MS
      value: "300000"
```

Apply changes and verify.

After updating your values.yaml, apply the changes with Helm:

helm upgrade <release-name> <chart> \
  -f values.yaml \
  --namespace <your-namespace>

Alternatively, for rapid testing without a full Helm upgrade, you can patch the deployment directly:

# Patch GMS deployment
kubectl set env deployment/<gms-deployment-name> \
  SEARCH_VALIDATION_MAX_QUERY_LENGTH=2000 \
  -n <your-namespace>

# Patch executor coordinator deployment
kubectl set env deployment/<executor-coordinator-deployment-name> \
  KAFKA_PROPERTIES_MAX_POLL_INTERVAL_MS=300000 \
  -n <your-namespace>

Confirm the sweeper is processing actions successfully.
After pods restart, monitor the executor coordinator logs to verify the sweeper is now completing its cycles without 400 errors:
```
kubectl logs -f deployment/<executor-coordinator-deployment-name> \
  -n <your-namespace> | grep -E "Sweeper|400|Bad Request"
```
You should see output similar to the following when the fix is working:
```
Sweeper(XXXXXXXXXX): plan execution finished: 1 of 1 actions succeeded.
```
You should not see any further 400 Client Error: Bad Request lines in the sweeper output.
Re-trigger any stuck ingestion runs.
Ingestion runs that became stuck before the fix was applied will not automatically recover. Trigger a fresh ingestion run for each affected source. The new runs will be tracked correctly and transition to SUCCESS or ABORTED as expected.

Root Cause Detail

The executor coordinator sweeper job (which runs on a 60-second interval) calls get_recent_requests_for_sources() in graph.py, which constructs a raw Elasticsearch query_string expression concatenating all active ingestion source URNs into a single GET URL parameter. With five or more sources, this query string grows large enough to trigger GMS's SEARCH_VALIDATION_MAX_QUERY_LENGTH guard, returning a 400 response. Because the sweeper's _get_restart_actions() method depends on the result of this query, every sweeper cycle fails entirely — no ingestion status transitions are processed, leaving all RUNNING jobs permanently stuck. This issue was fixed architecturally in a later release by refactoring the query to use the v3 entity scroll API with structured filters instead of raw Elasticsearch query strings. The SEARCH_VALIDATION_MAX_QUERY_LENGTH configuration change is the recommended mitigation for earlier releases.

Additional Notes

This issue is most likely to manifest when five or more ingestion sources are scheduled to run concurrently and share the same executor coordinator.
The SEARCH_VALIDATION_MAX_QUERY_LENGTH workaround is specifically required for versions of the executor coordinator that use raw Elasticsearch query strings in the sweeper (observed in executor v0.3.16.x). Later releases refactor this logic and do not require the workaround.
The KAFKA_PROPERTIES_MAX_POLL_INTERVAL_MS setting addresses a separate but co-occurring issue. Even after fixing the query length problem, high-parallelism ingestion scenarios benefit from the extended poll interval to prevent Kafka consumer group evictions under load.
The error RemoteExecutorStatus is missing required field: executorPoolId seen in sweeper logs is a separate, lower-severity warning related to stale executor registration records from previously terminated executor pods. It does not directly cause ingestion jobs to get stuck.
The MetadataAuditEvent_v4: Unknown topic or partition Kafka error seen in some log excerpts is expected in deployments that have migrated to v5 Kafka topics and can be safely ignored.
Upgrading to the latest available patch release is strongly recommended as it includes architectural fixes to the sweeper query logic that eliminate the root cause entirely.