Remote Executor Troubleshooting: Pool Assignment, Secret Injection, and Image Version Compatibility – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Remote Executor Configuration (AWS ECS / Fargate)

Issue

When deploying DataHub remote executors on AWS ECS/Fargate, operators may encounter several related problems: executors do not appear in the Manage Data Sources → Executor Pools UI, AWS Secrets Manager secrets fail to resolve in ingestion recipes (showing registered 0 secret(s)), and the executor container fails to start after upgrading to a newer image tag. These issues commonly stem from a combination of incorrect pool assignment environment variables, misconfigured or missing AWS Secrets Manager integration, and using an executor image version that is incompatible with the deployed DataHub Cloud instance version.

Error Messages

Discovery: failed to sending status to GMS: <error>
Discovery disabled: backend is not discovery-capable.
Secret masking enabled: registered 0 secret(s)
Can't connect to MySQL server on 'localhost' ([Errno 111] Connection refused)
dockerize: command not found (or equivalent startup failure after image upgrade)
HTTP 401 when calling the GMS GraphQL endpoint

You Might Be Asking

Why does the Manage Data Sources UI show no executor status even though the executor container is running?
Should I use DATAHUB_EXECUTOR_WORKER_ID or DATAHUB_EXECUTOR_POOL_ID to assign an executor to a pool?
Why are my AWS Secrets Manager secrets not being substituted into my ingestion recipe?
What executor image tag should I use for my version of DataHub Cloud?
Do I need to pass secret ARNs, or does the executor discover secrets automatically by name?

Solution

1. Verify the Correct Pool Assignment Environment Variable

The environment variable used to assign a remote executor to a pool differs by deployment method:

Docker deployments use DATAHUB_EXECUTOR_POOL_ID.
AWS CloudFormation template deployments map the ExecutorPoolId parameter to DATAHUB_EXECUTOR_WORKER_ID.
Custom ECS/Terraform deployments should use DATAHUB_EXECUTOR_WORKER_ID when following the CloudFormation-derived pattern.

The value must exactly match (case-sensitive) the pool name defined in DataHub Cloud under Manage Data Sources → Executor Pools. A single character difference (e.g., a hyphen vs. underscore) will prevent registration.

# Docker example
docker run -it \
  --env DATAHUB_GMS_URL="https://<your-instance>.acryl.io/gms" \
  --env DATAHUB_GMS_TOKEN="<your-pat-token>" \
  --env DATAHUB_EXECUTOR_POOL_ID="<pool-name-exactly-as-in-ui>" \
  --env DATAHUB_EXECUTOR_MODE=worker \
  --name datahub-executor-worker \
  <executor-image>:<version-tag>

# ECS task definition environment block (Terraform/CloudFormation-derived)
{
  "name": "DATAHUB_EXECUTOR_WORKER_ID",
  "value": "<pool-name-exactly-as-in-ui>"
}

2. Use the Executor Image Version That Matches Your DataHub Cloud Instance

The executor image tag must match the version of your DataHub Cloud instance. Using a mismatched tag (e.g., an old v0.3.x-acryl image against a v1.x.x-cloud instance) will cause startup failures, missing binaries (such as dockerize), and incompatible behavior.

Check your DataHub Cloud instance version in the UI or by contacting DataHub Support.
Use the corresponding image tag. For example, if your instance is on v1.1.0-cloud, use the v1.1.0-cloud tag (or the latest compatible patch, such as v1.1.3-cloud).
Note that newer image versions may have removed tools (e.g., dockerize, apt-get) that were present in older images. Audit any custom entrypoint commands in your task definition after upgrading.

# Example: correct image tag aligned to instance version
image: "<ecr-registry-url>/datahub-executor:v1.1.0-cloud"

3. Configure AWS Secrets Manager Secret Resolution (Runtime Mode)

The remote executor supports two distinct mechanisms for consuming AWS Secrets Manager secrets. Understanding which mode is active is critical.

Mode A — Runtime Secret Discovery (recommended for recipe variable substitution): The executor resolves ${SECRET_NAME} placeholders in recipes at runtime by looking up secrets named <prefix>SECRET_NAME in Secrets Manager. No ARNs are passed. This mode is disabled by default and must be explicitly enabled.

Enable the integration on the executor container:

DATAHUB_EXECUTOR_AWS_SM_ENABLED=true
DATAHUB_EXECUTOR_AWS_SM_REGION=<aws-region>
DATAHUB_EXECUTOR_AWS_SM_PREFIX=datahub-

Name your secrets in AWS Secrets Manager using the prefix. With the default prefix datahub-, a recipe reference of ${my_db_credentials} resolves to a secret named datahub-my_db_credentials. To use no prefix, set DATAHUB_EXECUTOR_AWS_SM_PREFIX to an empty string.

Grant the ECS task role permission to read secrets using the BatchGetSecretValue action:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:BatchGetSecretValue",
    "secretsmanager:ListSecrets",
    "secretsmanager:GetSecretValue"
  ],
  "Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:datahub-*"
}

Secrets are cached for 6 hours by default. To reduce the TTL (useful during testing), set:
```
DATAHUB_EXECUTOR_SECRET_CACHE_TTL=120
```
Confirm the integration is active at startup by checking executor logs for a line similar to:
```
Secret masking enabled: registered N secret(s)
```
If you see registered 0 secret(s), verify that the environment variables above are set on the deployed task definition (not just in source code), and that the secret names in AWS match the prefix+variable pattern exactly.

Mode B — Deploy-time ECS Secrets Injection: Secrets are injected as environment variables at task startup via the secrets block in the ECS task definition, referencing secret ARNs via valueFrom. This mode does not require enabling the runtime integration. Use this for credentials that the executor itself needs (e.g., DATAHUB_GMS_TOKEN), not for substituting values inside ingestion recipes.

# ECS task definition secrets block (deploy-time injection)
"secrets": [
  {
    "name": "DATAHUB_GMS_TOKEN",
    "valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:<your-token-secret-name>"
  }
]

The two modes are independent. For recipe variable substitution, always use Mode A (runtime discovery). For executor authentication credentials, Mode B (ARN injection) is appropriate.

4. Validate Executor Connectivity and Registration

Confirm the GMS endpoint is reachable from within the executor's network:

curl -sS -o /dev/null -w "%{http_code}\n" \
  https://<your-instance>.acryl.io/gms/health

Expected: 200

Confirm the GMS token is valid and has permission to call the GraphQL API:

curl -sS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer <DATAHUB_GMS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"query":"{ __typename }"}' \
  https://<your-instance>.acryl.io/gms/api/graphql

Expected: 200. A 401 indicates an invalid or expired token, or that the token does not have sufficient permissions.

Review executor container startup logs (not sub-process ingestion logs) for Discovery: lines indicating registration status.
Confirm that the pool is in READY state in Manage Data Sources → Executor Pools. A pool stuck in PROVISIONING_PENDING or PROVISIONING_IN_PROGRESS requires backend investigation by DataHub Support.

5. Use the Official Terraform Module (Strongly Recommended)

Manually converting the CloudFormation template to Terraform is error-prone and can silently omit critical configuration such as IAM grants, secrets injection, and correct environment variable mappings. DataHub maintains an official Terraform module that handles all of these correctly:

module "datahub_remote_executor" {
  source = "github.com/acryldata/datahub-terraform-modules//remote-ingestion-executor"

  datahub = {
    url              = "https://<your-instance>.acryl.io/gms"
    executor_pool_id = "<pool-name-exactly-as-in-ui>"
  }

  # Grant ECS execution role permission to read these secrets
  task_exec_secret_arns = [
    aws_secretsmanager_secret.datahub_access_token.arn,
    aws_secretsmanager_secret.my_db_credentials.arn,
  ]

  # Inject secrets as environment variables at task startup
  secrets = [
    {
      name      = "DATAHUB_GMS_TOKEN"
      valueFrom = aws_secretsmanager_secret.datahub_access_token.arn
    },
  ]
}

Both task_exec_secret_arns and secrets must be populated together. The first grants IAM permission; the second injects the value. If either is missing, secrets silently go unresolved.

Additional Notes

The executor image versioning scheme changed between older releases (v0.3.x-acryl) and current releases (v1.x.x-cloud). The newer images do not include tools such as dockerize or apt-get in the entrypoint. If your ECS task definition command block uses dockerize, you must update it when migrating to the v1.x.x-cloud image family. Always verify which image tag is recommended for your specific DataHub Cloud instance version with DataHub Support before upgrading. The executor discovery heartbeat interval is configurable via DATAHUB_EXECUTOR_DISCOVERY_INTERVAL (default: 30 seconds). Rotated secrets are not picked up until the cache TTL expires (default: 6 hours, controlled by DATAHUB_EXECUTOR_SECRET_CACHE_TTL in seconds); restart the ECS task to force an immediate refresh.