Remote Executor Ingestion Stuck in Pending: Diagnosing SQS Connectivity and Worker Process Issues – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Remote Executor / Queue Processing

Issue

An ingestion source remains indefinitely in a Pending state with no output and no visible errors. DataHub has successfully queued the run for a remote executor pool, but no executor picks it up. The remote executor may appear healthy — reporting a recent heartbeat in the UI — yet still fail to consume new tasks from the Amazon SQS queue. This is because the heartbeat mechanism and the task-consumer process use separate connections; a live heartbeat does not guarantee the worker process is functioning correctly.

Error Messages

No executor has picked up the run since [timestamp]
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sqs.<region>.amazonaws.com/"
Connection timed out (in executor logs referencing SQS or downstream data source)

You Might Be Asking

Why does my ingestion run stay in Pending even though the executor shows a recent heartbeat?
How can the executor appear alive but not process any queued runs?
What is the difference between the executor heartbeat and the SQS task consumer?
How do I confirm whether my remote executor can reach the SQS queue?
Will restarting the executor pod fix a stuck Pending run?

Solution

Step 1: Identify Which Executor Pool Is Assigned to the Source

In the DataHub UI, navigate to Ingestion > Sources and open the affected source.
Click Advanced and note the Executor ID (also called the pool ID).
If the pool is the DataHub-hosted default pool, contact DataHub Support to investigate the hosted executor and its queue.
If the pool is a customer-managed remote executor, continue with the steps below.

Step 2: Verify SQS Connectivity from Inside the Executor Pod

The executor must be able to reach the Amazon SQS endpoint to receive new run requests. Run the following command from inside the executor pod:

curl -v https://sqs.<your-aws-region>.amazonaws.com

Any HTTP response (including 403) — the host can reach SQS. Connectivity is not the problem; proceed to Step 3.
A hang, timeout, or refused connection — a proxy, firewall, or network policy is blocking outbound access to SQS. Investigate any network changes made since the executor last successfully processed a run, and ensure the following endpoints are reachable on port 443:
- sqs.<your-aws-region>.amazonaws.com
- pypi.org and files.pythonhosted.org (required for plugin installation at runtime)
- Your DataHub instance URL (https://<your-instance>.datahubproject.io)

Step 3: Review Executor Pod Logs

Retrieve executor pod logs starting from the time the run first entered the Pending state:

kubectl logs <executor-pod-name> -n <your-namespace> --since-time="<YYYY-MM-DDTHH:MM:SSZ>"

Search the logs for errors referencing SQS, botocore, or any network-related exceptions.
If the logs show no errors but the run is still Pending, the worker process inside the pod may have silently stalled.

Step 4: Restart the Remote Executor Pod

A rolling restart of the executor pod resolves the majority of cases where the heartbeat is active but the worker process has stopped consuming queue messages.

# For Kubernetes deployments
kubectl rollout restart deployment/<executor-deployment-name> -n <your-namespace>

After the restart, monitor the affected ingestion run in Ingestion > Sources. It should transition from Pending to Running within a few minutes if SQS connectivity is intact.

Step 5: Cancel Stuck Runs and Re-trigger if Needed

If the run does not move to Running after the restart, cancel it manually from the DataHub UI.
Trigger the source again to create a fresh run request.
Note: Triggering a source that already has a Pending run will mark one of the runs as Duplicate. This is expected behavior.

Additional Diagnostic Checks

Pool ID mismatch: Confirm the pool ID configured on the executor matches the Executor ID shown in the DataHub UI. Helm chart versions after 0.3.15 renamed worker_id to pool_id — a mismatch will prevent run pickup.
Version mismatch: Ensure the remote executor image version matches the DataHub server version. Running an outdated executor image is a known cause of stuck runs.
Capacity limits: A single executor pod can process a maximum of 4 concurrent ingestion runs. Additional runs will queue as Pending until a slot is available. If this is a recurring capacity issue, increase the number of executor replicas.
Pod crash or OOM: Check whether the executor pod was restarted recently due to an out-of-memory event. A crashed pod leaves in-flight runs in Pending state until the coordinator reassigns them.
```
kubectl describe pod <executor-pod-name> -n <your-namespace>
```

Additional Notes

The executor heartbeat and the SQS task-consumer process are independent. A healthy heartbeat visible in Ingestion > Executors does not confirm that the executor is consuming new run requests from the queue. Always verify SQS connectivity and worker process health separately. If outbound access to PyPI (pypi.org, files.pythonhosted.org) is blocked by a proxy or firewall, ingestion runs will silently return to Pending with no log output, as the executor cannot download required packages. This is a common but easily overlooked cause in environments with strict egress controls. When collecting information to escalate to DataHub Support, provide: the Executor ID of the affected source, executor pod logs from the time the run entered Pending, and details of any recent changes to network configuration, proxy settings, or executor version.