Skip to main content

Setup Walkthrough: Deploy DataHub Remote Executor (AWS ECS) via AWS CLI

Avery
Updated

This is an in-depth walkthrough of the process described here to setup the Datahub remote executor in AWS via the AWS CLI for DevOps engineers that are not intimately familiar with Datahub and for DataHub Admins that do not have the necessary access in AWS to deploy resources.

Permissions Checklist

Critical: The person running the final command must have the following AWS permissions.

  • For the DataHub User: Do not try to run this if you don't have AWS access. Send this guide to your DevOps or Cloud Infrastructure team.
  • For the AWS Admin: The script uses CloudFormation to provision resources. Your IAM user/role needs Allow permissions for:
    • CloudFormation: CreateStack, DescribeStacks
    • ECS: CreateCluster, RegisterTaskDefinition, CreateService
    • IAM: CreateRole, AttachRolePolicy, PassRole (Required to create the execution role for the ECS task).
    • EC2: DescribeVpcs, DescribeSubnets, CreateSecurityGroup
    • CloudWatch Logs: CreateLogGroup

Step 1: Get your DataHub Connection Details

Skip this if you are an AWS Admin—ask your DataHub user to provide these three strings.

1. The Datahub GMS URL

  • Copy your Datahub URL from the browser.
  • Add /gms to the end.
  • Example: https://mycompany.acryl.io/gms

2. The Password (Access Token)

  • In DataHub, click Settings (Gear Icon) (top right) > Access Tokens.
  • Click Generate new token.
  • Select Type: Remote Executor.
  • Copy the long text string.

3. The Pool Name

  • In DataHub, click Data Sources (left sidebar) > Executors.
  • Click Create new Pool.
  • Name it: remote (case sensitive).

Step 2: Get your AWS Network Details

Skip this if you are a DataHub user—ask your AWS Admin to provide these three details. 

Run these commands in your terminal to automatically find the correct settings for your environment.

1. Get the AWS Region

aws configure get region

(Save the output. It looks like us-east-1 or us-west-2)

2. Get the VPC ID (Network)

aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query "Vpcs[0].VpcId" --output text

(Save the output. It looks like vpc-12345abc)

3. Get the Subnet ID (Lane)
Replace YOUR_VPC_ID below with the result from command #2.

aws ec2 describe-subnets --filters Name=vpc-id,Values=YOUR_VPC_ID --query "Subnets[0].SubnetId" --output text

(Save the output. It looks like subnet-6789xyz)

Step 3: Download the Template

You must download the configuration file to your computer before running the installer.

  1. Open your terminal/command prompt.
  2. Run this command to download the file to your current folder:
curl -O https://raw.githubusercontent.com/acryldata/datahub-cloudformation/master/remote-executor/datahub-executor.ecs.template.yaml

Note: If you prefer, you can download the datahub-executor.ecs.template.yaml template file manually from your browser, but ensure you save it in the same folder where you are running your terminal commands.

Step 4: Run the Deployment

We will pass all your details directly into the AWS CLI.

  1. Copy the code block below into a text editor (like Notepad).
  2. Replace the 6 values at the top with your specific details from Steps 1 & 2.
  3. Paste the entire block into your terminal and hit Enter.
# ==========================================
# CONFIGURATION - FILL THIS SECTION OUT
# ==========================================

# 1. Your DataHub URL (MUST end in /gms)
DH_URL="https://mycompany.acryl.io/gms"

# 2. Your DataHub Access Token
DH_TOKEN="eyJhbGciOiJIUz..."

# 3. Your AWS Region (from Step 2.1)
MY_REGION="us-west-2"

# 4. Your AWS VPC ID (from Step 2.2)
MY_VPC="vpc-12345abc"

# 5. Your AWS Subnet ID (from Step 2.3)
MY_SUBNET="subnet-6789xyz"

# 6. Your Pool Name (Usually 'remote')
POOL_ID="remote"

# ==========================================
# DEPLOYMENT COMMAND
# ==========================================

aws --region $MY_REGION cloudformation create-stack \
  --stack-name datahub-remote-executor \
  --template-url https://raw.githubusercontent.com/acryldata/datahub-cloudformation/master/remote-executor/datahub-executor.ecs.template.yaml \
  --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM \
  --parameters \
    ParameterKey=ExecutorPoolId,ParameterValue="$POOL_ID" \
    ParameterKey=VPCID,ParameterValue="$MY_VPC" \
    ParameterKey=SubnetID,ParameterValue="$MY_SUBNET" \
    ParameterKey=DataHubBaseUrl,ParameterValue="$DH_URL" \
    ParameterKey=DataHubAccessToken,ParameterValue="$DH_TOKEN"

Step 5: Verify it worked

Wait 5 minutes, then run this command to check the status.
(Make sure to replace YOUR_REGION with the region you found in Step 2).

aws cloudformation describe-stacks --stack-name datahub-remote-executor --region YOUR_REGION --query "Stacks[0].StackStatus"
  • Result: CREATE_COMPLETE
    • Action: You are done. Go to DataHub > Ingestion > Executors. You should see a green "Active" dot next to your pool.
  • Result: ROLLBACK_COMPLETE

    • Action: Deployment failed. Usually, this means the URL didn't end in /gms or permissions were insufficient. Run:

      aws cloudformation describe-stack-events --stack-name datahub-remote-executor --region YOUR_REGION to see the error.

Troubleshooting

GMS returned no executor configs

If you get the following error, verify that your parameters are specified correctly and that you have created the remote executor pool in Datahub.

Retrying datahub_executor.common.client.config.resolver.ExecutorConfigResolver._fetch_executor_configs in 16.0 seconds as it raised
Exception: GMS returned no executor configs, unable to proceed..

Connection Failed

  • Verify network connectivity
  • Check DataHub URL configuration
  • Validate Executor Pool ID
  • Validate access token

Secret Access Failed

  • Confirm secret ARNs/names
  • Check permissions
  • Verify secret format

Container Failed to Start

  • Check resource limits
  • Verify registry access
  • Review container logs

Related to

Related articles