Ingestion Jobs Stuck in Pending Status Due to Remote Executor SQS Connectivity Issues – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Remote Executor / Executor Pool Connectivity

Issue

Ingestion datasource runs remain indefinitely in PENDING status for a specific remote executor pool, while runs routed to other executor pools or managed system tasks complete successfully. No logs appear in the DataHub UI for the affected runs because logs are only emitted after an executor worker claims a job — if nothing claims the job, it stays silently pending. Investigation reveals that the remote executor host is alive and sending heartbeats to DataHub Cloud, but is unable to poll its assigned AWS SQS queue for new work. This is typically triggered by a networking change — such as a security group rule update, NAT gateway change, routing table modification, or removal of an SQS VPC endpoint — that blocks outbound access from the executor host to the SQS service endpoint.

Error Messages

PENDING (run status in DataHub UI — no further error detail visible)
CANCELLED (run status after the 48-hour sweeper threshold elapses with no executor pickup)

You Might Be Asking

Why are my ingestion runs stuck in pending with no logs?
Why is only one of my executor pools affected while others work fine?
My remote executor is running and heartbeating — why isn't it picking up jobs?
How do I tell if my remote executor has lost access to SQS?
Why do pending runs eventually disappear or show as cancelled?

Solution

Confirm the executor is heartbeating but not consuming jobs.
A remote executor can be online and successfully sending heartbeats to DataHub Cloud while simultaneously being unable to pull messages from its SQS queue. These are two separate network paths. Heartbeat success does not imply SQS access.
Test outbound connectivity from the executor host to AWS SQS.
SSH into or exec into the executor host and run the following command. Any HTTP response (including a 403 Forbidden) confirms the network path is open. A connection timeout or refused connection indicates the egress path to SQS is blocked.
```
curl -v https://sqs.<your-aws-region>.amazonaws.com
```
If connectivity is blocked, check for recent changes to any of the following in the VPC hosting the executor:
- Security group outbound rules
- NAT gateway configuration or route tables
- SQS VPC endpoint presence and policy
- Network ACLs
Restore SQS connectivity using one of the following approaches:
- Add or restore an outbound security group rule allowing HTTPS (port 443) egress to the SQS service endpoint for your region.
- Ensure the NAT gateway is correctly configured and the route table for the executor's subnet includes a default route via the NAT gateway.
- Alternatively, create or restore an SQS VPC endpoint in the executor's VPC so traffic stays within the AWS network and does not require a NAT path:
```
# Example: Create an SQS interface VPC endpoint via AWS CLI
aws ec2 create-vpc-endpoint \
  --vpc-id <your-vpc-id> \
  --service-name com.amazonaws.<your-aws-region>.sqs \
  --vpc-endpoint-type Interface \
  --subnet-ids <your-subnet-id> \
  --security-group-ids <your-security-group-id>
```
Verify the executor pool ID matches the configured ingestion source.
If the executor was redeployed around the time runs started failing, confirm that the pool ID the executor registered with exactly matches the executorId value configured for your ingestion sources in the DataHub UI. A case-sensitive mismatch will cause all routed runs to remain unclaimed.

Check executor container logs for SQS-related errors.

On the executor host, retrieve the container or process logs and search for SQS connection errors:

# For Docker-based deployments
docker logs <executor-container-name> 2>&1 | grep -i sqs

# For Kubernetes-based deployments
kubectl logs -n <namespace> <executor-pod-name> | grep -i sqs

Manually cancel stuck pending runs.
Pending runs will not self-cancel for up to 48 hours (the default sweeper threshold). To unblock your pipeline immediately, cancel the stuck runs via the DataHub UI or API, then re-trigger them after SQS connectivity is restored.
Enable executor log delivery.
If executor logging is disabled in your deployment configuration, enable it so that future run failures produce visible logs in the DataHub UI. In your remote executor deployment configuration, set:
```
remote_executor_logging_enabled: true
```
Without this setting, even a healthy executor produces no log output in the UI, which makes diagnosing future issues significantly harder.

Additional Notes

This issue pattern — executor heartbeat healthy, SQS polling broken — is a repeatable class of failure that typically follows a networking change in the VPC hosting the remote executor. Common triggers include security group rule updates, NAT gateway replacements, or VPC endpoint deletions. Because the heartbeat path (HTTPS to DataHub Cloud GMS) and the job polling path (HTTPS to AWS SQS) are distinct, a change that only affects egress to SQS will leave heartbeat telemetry appearing healthy while all job pickup silently stops. The absence of logs in the DataHub UI is expected and by design when no executor claims a run — it is not itself a bug. Pending runs that are never claimed will be automatically transitioned to CANCELLED by the sweeper process after approximately 48 hours. DataHub Cloud manages all backend GMS and executor coordination infrastructure; SQS queue access from the remote executor runs in the customer's own AWS environment and is the customer's responsibility to maintain. If all networking checks pass and runs remain stuck, contact DataHub Support to inspect executor pool heartbeat status and backend queue health.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation