Multi-Cloud Ingestion: Connecting a Single DataHub Instance to Both AWS and Azure Sources – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Remote Executor / Multi-Cloud Configuration

Issue

Organizations running DataHub Cloud in a hybrid or multi-cloud environment may wonder whether a single DataHub UI instance can simultaneously connect to and ingest metadata from sources across multiple cloud providers — for example, Azure and AWS. A common scenario is having the DataHub control plane (UI + GMS) hosted in one cloud environment while needing to ingest from data catalogs and services in a separate cloud environment, such as AWS Glue, S3, Athena, or Redshift, without deploying a separate DataHub instance.

You Might Be Asking

Can a single DataHub UI instance connect to both AWS and Azure data sources simultaneously?
Do I need to deploy separate DataHub instances for each cloud provider?
How do I configure a remote executor in AWS when my DataHub control plane is hosted in Azure?
Can I have multiple remote executor pools pointing to the same DataHub Cloud instance?
Will adding an AWS remote executor affect my existing Azure ingestion sources?

Solution

This is fully supported. DataHub Cloud is designed to work with multiple remote executor pools across different cloud environments, all connecting to the same control plane (UI + GMS). You do not need separate DataHub instances — one UI can orchestrate ingestion from both Azure and AWS simultaneously. Connectivity is configured per ingestion source, not per DataHub instance, so existing sources in one cloud are unaffected when you add sources from another cloud.

Understand the architecture: Each cloud environment gets its own Remote Executor Pool with a unique Pool ID. Your existing cloud setup (e.g., Azure) already uses one pool. You register your second cloud's executor (e.g., AWS) as an additional pool pointing to the same DataHub Cloud instance.
Generate a Remote Executor Access Token: In the DataHub UI, navigate to Settings → Access Tokens and generate a new token. This token will be used by the AWS-deployed remote executor to authenticate with your DataHub Cloud instance.
Deploy the remote executor in your AWS environment using one of the following supported methods:
- ECS/Fargate: Use the provided CloudFormation template.
- EKS: Use the Helm chart from the DataHub executor Helm repository.
- Terraform: Use the DataHub Terraform modules for remote ingestion executor.

Configure the AWS remote executor with the following key environment variables:

# URL of your DataHub Cloud GMS endpoint
DATAHUB_GMS_URL=https://<your-instance>.acryl.io/gms

# Access token generated in Step 2
DATAHUB_GMS_TOKEN=<your-access-token>

# A unique identifier for this executor pool (must differ from your existing pool)
DATAHUB_EXECUTOR_POOL_ID=aws-<environment-name>

Ensure the pool ID is unique and does not conflict with your existing Azure executor pool ID (e.g., default or another name already in use).

Grant AWS credentials to the executor so it can access AWS data sources (Glue, S3, Athena, Redshift, etc.). Credentials can be provided via any of the following methods:
- IAM role attached to the ECS task, EC2 instance, or EKS service account running the executor (recommended).
- Environment variables or secrets injected into the executor deployment.
Verify network connectivity: The AWS remote executor requires outbound connectivity only — it polls and connects out to your DataHub Cloud URL. No inbound ports need to be opened on the executor side.
```
# Confirm outbound connectivity from the executor host to your DataHub Cloud endpoint
curl -I https://<your-instance>.acryl.io/gms/health
# Expected: HTTP 200 OK
```
Create ingestion sources in the DataHub UI for your AWS data catalogs. When configuring each new ingestion source, a dropdown will appear allowing you to select the executor pool to use. Select the AWS executor pool ID you configured in Step 4. Your existing Azure ingestion sources will continue to use their original executor pool and do not need to be modified.

Additional Notes

This multi-cloud executor pattern is supported for DataHub Cloud (managed) deployments. For self-hosted DataHub, remote executor functionality may differ — consult the open-source documentation for your version.
Each remote executor pool must have a globally unique DATAHUB_EXECUTOR_POOL_ID within your DataHub Cloud instance. Reusing an existing pool ID will cause conflicts.
Credentials and network access remain scoped within each cloud environment. The DataHub Cloud control plane does not require direct access to your cloud data sources — only the remote executor in that environment does.
Adding a new remote executor pool in AWS has no impact on existing ingestion sources configured to use other executor pools (e.g., Azure). Each ingestion source is independently associated with a specific executor pool.
When using EKS, ensure the Kubernetes service account has the appropriate IAM role binding (via IRSA — IAM Roles for Service Accounts) to access AWS services without static credentials.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation