Skip to main content

Remote Executor Ingestion Jobs Stuck in Pending Status

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Remote Executor Task Processing

Issue

Remote executor ingestion jobs remain stuck in "Pending" status despite healthy executor instances showing fresh monitoring and discovery logs. The executors appear to be running normally but fail to pick up queued ingestion tasks from SQS, causing ingestions to never progress beyond the pending state.

Error Messages

  • Scheduled X 'ingestion-sources' execution_requests! - repeated in coordinator logs without task pickup
  • ERROR: Failed to deserialize executor record - sweeper unable to clean up stale executor records
  • UNKNOWN_TOPIC_OR_PARTITION - executor cannot find required Kafka topics

You Might Be Asking

  • Why do my executors show healthy logs but don't process ingestion tasks?
  • Why does restarting the executor pod fix the issue temporarily?
  • How can I prevent ingestion jobs from getting stuck in pending status?

Solution

The issue typically stems from one or more of these root causes:

1. Aggressive Restart Automation Interfering with SQS

If you have automated restarts based on log silence, disable them temporarily as they can create SQS visibility timeout loops:

  1. Disable any Lambda functions or scripts that restart executors based on log patterns
  2. The monitoring/discovery logs and ingestion task pickup are separate mechanisms
  3. Restart automation may kill executors before they can process SQS messages, causing visibility timeouts

2. Tune Executor Concurrency Settings

Reduce concurrent job execution to prevent memory exhaustion:

DATAHUB_EXECUTOR_INGESTION_MAX_WORKERS=2

For heavy ingestion sources, increase task weight in the source configuration:

{"EXECUTOR_TASK_WEIGHT": "0.8"}

3. Verify Executor Pool Configuration

  1. Confirm the DATAHUB_EXECUTOR_POOL_ID environment variable on executor instances matches the pool ID configured in DataHub
  2. Check that tasks are being routed to the correct SQS queue
  3. If the executor pool was recreated during scaling, cancel stuck executions and retrigger them

4. Check SQS Queue State

In the AWS Console, examine your executor pool's SQS queue:

  1. Look for high "Messages In Flight" counts (indicates stuck processing)
  2. Check "Messages Available" for queued tasks
  3. Review Dead Letter Queue depth if configured

5. Clean Up Stale Executor Records

For on-premise deployments, stale executor records can disrupt the coordinator:

  1. Identify executor records missing required fields like executorPoolId
  2. Delete these records via the DataHub API
  3. Restart the coordinator to refresh executor slot accounting

6. Upgrade DataHub SDK

For Teradata and other memory-intensive sources, upgrade to SDK v1.5.0.17 or later to fix memory leaks that can cause executor crashes.

Additional Notes

Restarting executor pods is the correct recovery procedure when jobs are stuck. The coordinator will detect stale executors after the configured timeout and reassign pending tasks to healthy instances. For DataHub Cloud customers, ensure your executor instances have adequate memory and CPU resources. For on-premise deployments, verify all required Kafka topics exist and are accessible by the executor.

Related Documentation

Tags: remote-executor, pending-ingestion, sqs-queue, executor-pool, task-processing, memory-management, coordinator, sweeper, visibility-timeout, ingestion-stuck

Related articles