Remote Executor Security Hardening: Restricting Environment Variable Access and GMS Token Exposure – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Remote Executor Security Configuration

Issue

Self-hosted Remote Executor deployments (on Kubernetes environments such as GKE or EKS) expose a security risk where any user with permission to edit an ingestion source can potentially read or exfiltrate sensitive pod-level environment variables — including DATAHUB_GMS_TOKEN. Two distinct code paths create this exposure: (1) SubProcessRecipeTaskArgs.get_combined_env_vars() merges extra_env_vars directly over os.environ with no allowlist or blocklist, allowing interpreter-level variables such as PYTHONPATH and LD_PRELOAD to influence execution before the recipe is evaluated; and (2) SubProcessTaskUtil._resolve_recipe() falls back to os.environ when a ${PLACEHOLDER} is not resolved by any configured secret store, and the same fallback exists in _create_secret_stores() for assertion and monitor connections where EnvironmentSecretStore is instantiated first. The combined effect is that a recipe author can embed ${DATAHUB_GMS_TOKEN} in a connection field and cause the token to be transmitted to an arbitrary host specified in the recipe.

Error Messages

No explicit error is raised; the exposure is silent data exfiltration via crafted recipe placeholders.

You Might Be Asking

Can I configure the Remote Executor to ignore or allowlist extra_env_vars so ingestion sources cannot inject dangerous variables?
Can I disable the fallback to os.environ when resolving ${PLACEHOLDER} values in recipes, on both the ingestion and assertion/monitor paths?
Is there a supported way to supply the GMS token to the executor as a file instead of an environment variable, so it is not reachable via os.environ?
Does filtering extra_env_vars or the env fallback fully prevent a malicious ingestion source editor from accessing the GMS token?

Solution

Apply the following mitigations in combination. No single control is sufficient on its own.

Restrict who can edit ingestion sources (primary control).
The fundamental trust boundary is the Manage Metadata Ingestion platform privilege. A user who can edit an ingestion source can already execute arbitrary code on the executor pod — for example, by switching a source off the bundled CLI version and adding pip libraries that install at run time. Code running in that context can read the GMS token regardless of where it is stored. Filtering extra_env_vars or disabling the env fallback reduces attack surface but does not close the underlying privilege boundary.

In DataHub, navigate to Settings → Permissions → Roles and ensure that the Manage Metadata Ingestion privilege is granted only to users you would trust with direct access to the executor pod.
Mount the GMS token as a file instead of an environment variable (v2.3.1-cloud / chart 0.0.72+).
Starting with DataHub Cloud chart version 0.0.72 (executor image v2.3.1-cloud), you can enable token-file mode. When enabled, the Helm chart mounts your GMS token secret read-only at /etc/datahub/auth/token and stops setting the DATAHUB_GMS_TOKEN environment variable entirely. This breaks the ${DATAHUB_GMS_TOKEN} exfiltration path because the variable no longer exists in os.environ.

Add the following override to your Helm values file:
```
global:
  datahub:
    auth:
      type: pat
      tokenFileEnabled: true
```
The chart will read the token from the secret referenced by gms.secretRef / gms.secretKey in your existing values and mount it at the path above.

Important: Do not place the token file inside a directory already mounted as a general secret store (e.g., /mnt/secrets). Recipes can read arbitrary files from paths accessible to the secret store, so a dedicated mount path outside /mnt/secrets is required.

Apply a Kubernetes NetworkPolicy to restrict executor pod egress.

Add a NetworkPolicy that limits egress from executor pods to known, trusted DataHub endpoints and blocks access to public package indexes (PyPI, etc.). If your deployment uses only bundled virtual environments, no legitimate workload requires outbound access to package indexes. Blocking them also prevents a malicious source from installing additional packages at run time.

Example NetworkPolicy skeleton (adapt selectors and CIDRs to your environment):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: executor-egress-restrict
  namespace: <your-executor-namespace>
spec:
  podSelector:
    matchLabels:
      app: datahub-executor   # adjust to match your executor pod label
  policyTypes:
    - Egress
  egress:
    # Allow traffic to your DataHub GMS / frontend endpoints only
    - to:
        - ipBlock:
            cidr: <your-datahub-gms-cidr>/32
      ports:
        - protocol: TCP
          port: 8080
    # Allow DNS resolution
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
    # Block everything else (implicit deny when NetworkPolicy is applied)

Track pending platform-level controls.
The following executor configuration flags do not yet exist as of acryl-executor 0.3.19 / v2.2.0-cloud but have been identified as roadmap items:
- DATAHUB_EXECUTOR_DISABLE_ENV_FALLBACK_RESOLUTION — disables the os.environ fallback in _resolve_recipe() and _create_secret_stores() for both ingestion and assertion/monitor paths.
- DATAHUB_EXECUTOR_EXTRA_ENV_VAR_ALLOWLIST (or blocklist) — filters which keys in extra_env_vars are merged into the subprocess environment via get_combined_env_vars().
- DATAHUB_EXECUTOR_ENVIRONMENT_SECRET_STORE_ENABLED=false — removes EnvironmentSecretStore from the secret resolution chain entirely.
Until these flags are available, the controls in steps 1–3 above provide the most effective defense-in-depth posture. If you require these flags urgently, open a support ticket or feature request so engineering can prioritize the work.

Additional Notes

The token-file mount feature (tokenFileEnabled: true) requires chart version 0.0.72 or later and executor image v2.3.1-cloud or later. It is not available in v2.2.0-cloud (acryl-executor 0.3.19). If you are on an earlier version, upgrading to v2.3.1-cloud is a prerequisite for the file-based token approach. Environments running in staging can be used to validate the Helm override before applying it to production. The egress NetworkPolicy example is a template only — CIDRs, namespace selectors, and port numbers must be adapted to match your specific cluster topology and DataHub deployment addresses. The os.environ fallback and the extra_env_vars merge behavior described in this article reflect the codebase as of acryl-executor 0.3.19 / CLI 1.7.0.3; behavior may change in future releases as the roadmap controls are implemented.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation