Remote Executor Bundled venv and Locked Image Security Hardening Best Practices – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Remote Executor Security Hardening & Bundled Ingestion venvs

Issue

Organizations performing security reviews of their Remote Executor deployments — particularly those concerned about supply chain attack risks from runtime PyPI package installation — often have questions about how bundled virtual environments (venvs) work, when they are used, the limitations of locked images, the distinction between image environment variables and build arguments, and how to extend images safely. This article consolidates guidance on all of these topics to help teams make informed decisions when hardening their Remote Executor setup.

You Might Be Asking

Does deployment.cli_version need to be explicitly set for a bundled venv to be used, or is it picked up automatically?
Can I use dev, RC, or patch DataHub CLI versions with bundled venvs or locked images?
Why can't I use BUNDLED_CLI_VERSION or BUNDLED_VENV_PLUGINS as environment variables when extending images?
Can I extend a locked image to add new connectors or plugins?
What do I lose by switching to a fully locked image for my Remote Executor?
What happens if the bundled CLI version doesn't match managedIngestion.defaultCliVersion in GMS?
Can I use a custom or private source package with a bundled venv?

Solution

1. When Are Bundled venvs Used?

The executor selects a venv using the following resolution order:

Each recipe may optionally specify deployment.cli_version. If omitted, the executor falls back to managedIngestion.defaultCliVersion configured in GMS.
The executor compares the resolved version string against what is installed in the pre-built bundled venv.
If the version matches, the pre-built bundled venv is used. If no match is found, the executor falls back to an on-demand pip install at runtime (on non-locked images).

To guarantee that bundled venvs are always used — for example, in air-gapped or security-hardened environments — set the CLI version to the literal string "bundled" either globally in GMS or per recipe:

# Per-recipe example
deployment:
  cli_version: "bundled"

When defaultCliVersion is set to "bundled" at the GMS level, all recipes that do not override deployment.cli_version will attempt to use a bundled venv. If no bundle exists for a given source, ingestion will error rather than fall back to runtime installation.

2. One CLI Version Per Image

Each image is built with a single CLI version (BUNDLED_CLI_VERSION) at build time. This means:

All bundled venvs within a given image share the same DataHub CLI version.
Using dev wheels, RC builds, or patch releases of the DataHub CLI requires either runtime package installation (not available on locked images) or building a custom image with the desired version pre-bundled.
If your workflows depend on multiple CLI versions simultaneously, locked images are not suitable without maintaining separate custom images per version.

3. Environment Variables vs. Build Arguments

A common source of confusion when extending Remote Executor images is the distinction between image ENV variables and Docker build arguments (ARG):

DATAHUB_BUNDLED_VENV_PATH is an ENV variable and is available at runtime inside the container.
BUNDLED_CLI_VERSION and BUNDLED_VENV_PLUGINS are build arguments (ARG), not environment variables. They are consumed only at image build time and are not persisted in the final image.

This means you cannot read or rely on BUNDLED_CLI_VERSION or BUNDLED_VENV_PLUGINS at container runtime. When extending a non-locked base image, pass these as --build-arg values during your Docker build:

docker build \
  --build-arg BUNDLED_CLI_VERSION=1.6.0 \
  --build-arg BUNDLED_VENV_PLUGINS="snowflake,bigquery,redshift" \
  -t my-org/datahub-ingestion:custom \
  -f Dockerfile .

4. Locked Images: Limitations and Customization Strategy

Locked images (e.g., *-cloud-locked variants) have pip, uv, and uvx binaries removed from all venvs via strip_pip_uv_from_venvs.sh, called by lock_image.sh. This prevents any runtime package installation but also means:

You cannot extend a locked image to add new bundles or plugins after the fact.
Locked images ship with a limited set of pre-built connectors (e.g., s3-bundled, file-bundled). Additional connectors must be baked in at build time.
Locked images are not officially supported as a fully productized workflow at this time (target: next major release). Use with caution in production.

The recommended approach for customizing a hardened image is to extend a non-locked base image (*-cloud or *-cloud-slim), build your additional bundles into it, and then optionally apply the locking step yourself:

FROM <your-registry>/datahub-ingestion:v2.0.0-cloud

# Build additional bundled venvs using the included helper script
RUN --mount=type=secret,id=pip_conf,dst=/etc/pip.conf \
    BUNDLED_CLI_VERSION=1.6.0 \
    BUNDLED_VENV_PLUGINS="snowflake,bigquery,mysql" \
    /usr/local/bin/build_bundled_venvs_unified.sh

# Optionally apply the lock (removes pip/uv from all venvs)
RUN /usr/local/bin/lock_image.sh

The helper scripts build_bundled_venvs_unified.sh, strip_pip_uv_from_venvs.sh, and lock_image.sh are included in /usr/local/bin of non-locked images (available in v2.0.0+). Building entirely from scratch is also possible but not necessary when these scripts are available.

5. Behavior When CLI Version Mismatches on Locked Images

When using a locked image, if the resolved CLI version (from recipe or GMS) does not match the version bundled in the image, ingestion will fail rather than fall back to runtime installation (since pip and uv are unavailable). To avoid this:

Set deployment.cli_version: "bundled" in every recipe, or set managedIngestion.defaultCliVersion to "bundled" in GMS.
After upgrading the image, the bundled venv continues to be used — there is no fallback to runtime installation on locked images, and no error as long as the version string is set to "bundled".

6. Extra pip Requirements and Custom Source Packages

If a recipe specifies extra_pip_requirements, those packages are installed into the venv at runtime. This is compatible with bundled venvs on non-locked images — the bundled venv is used as the base and the extra packages are added on top at runtime.

On locked images, extra_pip_requirements are not supported, as the package installers have been removed. Custom or private source packages intended for use with locked images must be pre-installed into a custom-built image at build time.

Note: Bundling venvs for fully custom source packages (not part of the standard DataHub connector set) is not yet supported as a first-class feature. As an interim approach, add your custom package as an extra_pip_requirements entry in the recipe — it does not need to be publicly available on PyPI:

source:
  type: my-custom-source
  config:
    ...

pipeline_name: my_pipeline

extra_pip_requirements:
  - "my-custom-datahub-source==1.0.0"
  - "--extra-index-url https://<your-private-registry>/simple/"

7. What You Lose With a Fully Locked Image

The following capabilities are unavailable when using a fully locked image:

Runtime override of the CLI version via deployment.cli_version (non-"bundled" values will fail)
extra_pip_requirements in recipes
Use of dev, RC, or patch CLI versions without rebuilding the image
Adding new connectors or plugins without rebuilding the image

In return, you gain the guarantee that no network-fetched packages are installed at runtime, significantly reducing exposure to supply chain attacks.

Additional Notes

Full official support for locked image workflows is targeted for a future major release. Consult the DataHub release notes for the latest status.
The helper scripts strip_pip_uv_from_venvs.sh and lock_image.sh are available in v2.0.0+ images. If you are on an older image and cannot find these scripts, upgrade to a v2.0.0 or later base image before attempting to apply the locking step manually.
If you are deploying a Remote Executor with strict security requirements, DataHub's security team is available to discuss your specific architecture. Contact your DataHub representative to arrange a security review call.
Credentials baked into older image builds for read-only package registry access were addressed in DataHub v2.0.0. Upgrade to v2.0.0 or later to ensure these credentials are no longer present.
Setting managedIngestion.defaultCliVersion to "bundled" globally will cause ingestion to error for any source that does not have a pre-built bundled venv in the image. Ensure all required connectors are bundled before applying this setting globally.