Skip to main content

Remote Executor Can Only Connect to a Single DataHub Instance; Separate Executors Required per Environment

Angelo Bresci
Updated

Area: Deployment Issues

Sub-Area: Remote Executor Configuration & Multi-Environment Architecture

Issue

Users attempting to reuse an existing production remote executor for a staging (or other secondary) DataHub environment find this is not possible. A remote executor deployment binds to exactly one DataHub instance at container startup via the DATAHUB_GMS_URL environment variable. There is no dynamic multi-environment routing, and a single executor cannot be shared or redirected to a different DataHub instance at runtime. Each environment — production, staging, development, etc. — requires its own independently configured and deployed remote executor. A common follow-on issue is that ingestion sources copied from a production environment retain their original executor pool assignments, causing them to fail in staging because those pools either do not exist or are inactive in the staging environment.

You Might Be Asking

  • Can I point my production remote executor at my staging DataHub instance?
  • Can one remote executor serve multiple DataHub environments?
  • Why are my staging ingestion sources failing after I copied them from production?
  • Why does my executor not appear in the staging DataHub UI after deployment?

Solution

Follow these steps to deploy a dedicated remote executor for each additional DataHub environment (e.g., staging).

  1. Create an Executor Pool in the target environment's DataHub UI.

    In your staging DataHub instance, navigate to Settings → Ingestion → Remote Executor Pools (or Data Sources → Executors → Create) and create a new pool. Note the pool ID — it must exactly match the pool identifier you configure in the executor deployment.

  2. Generate a new Personal Access Token (PAT) from the staging instance.

    Production tokens cannot be reused. In your staging DataHub UI, go to Settings → Access Tokens → Generate New Token and select the Remote Executor token type. Copy the generated token securely.

  3. Deploy a new remote executor instance pointing to the staging environment.

    Configure the following environment variables for the new executor deployment (ECS task, Kubernetes pod, or Docker container):

    # Required environment variables for a staging remote executor deployment

DATAHUB_GMS_URL=https://<your-staging-instance>.acryl.io/gms
DATAHUB_SYSTEM_CLIENT_ID=<remote-executor-client-id>
DATAHUB_SYSTEM_CLIENT_SECRET=<staging-personal-access-token>
EXECUTOR_POOL_ID=<staging-pool-id>

    Note: A mismatch between EXECUTOR_POOL_ID and the pool ID created in the DataHub UI is the most common reason an executor deploys successfully but does not appear in the Executors UI.

  4. Verify network connectivity from the new executor host.

    Ensure the executor deployment can reach the staging DataHub GMS endpoint and any required AWS SQS queues. If the staging environment enforces inbound IP allowlisting, add the outbound IP address of the new executor deployment to the staging allowlist.

  5. Start the executor and confirm registration.

    Once started, the executor should register with the staging environment and appear in Settings → Ingestion → Remote Executor Pools in the staging DataHub UI. Check pool health using the monitoring documentation linked below.

  6. Reassign ingestion sources that were copied from production.

    If ingestion sources were copied or migrated from a production environment, they may still reference production executor pool IDs that do not exist in staging. For each affected source, update the pool assignment:

    • Open the ingestion source in the staging DataHub UI.
    • Navigate to Advanced → Executor Pool.
    • Select the correct staging executor pool (e.g., the pool created in Step 1).
    • Save and re-trigger the ingestion run.

Additional Notes

  • The single-destination design of the remote executor is intentional. DATAHUB_GMS_URL is loaded once at startup; there is no runtime mechanism to route tasks to multiple DataHub instances from a single executor process.
  • Ingestion sources that were cloned or imported from another environment frequently carry over executor pool IDs from that source environment. Always audit pool assignments after any environment migration or copy operation.
  • If you are running DataHub version 2.0 with mTLS-enabled executor deployments, be aware that a task-dispatch bug (HTTP session not reused on poll requests) existed in v2.0 and was resolved in v2.1. Upgrade to v2.1 or later if you encounter task dispatch failures in an mTLS configuration.
  • Each environment's executor pool health can be monitored independently. Refer to the Remote Executor Monitoring documentation for details on checking pool status.

Related Documentation

Tags: remote-executor, multi-environment, staging, executor-pool, ingestion, deployment, configuration, environment-isolation, pool-id-mismatch, datahub-cloud