Offline Dependency Resolution for the Remote Ingestion Executor in Air-Gapped Environments – DataHub Customer Support Portal

```html

Area: Deployment Issues

Sub-Area: Remote Ingestion Executor (RIE) — Air-Gapped / Private Subnet Deployments

Issue

In environments where the Remote Ingestion Executor (RIE) cannot reach the public PyPI index at runtime — such as private subnets, air-gapped networks, or environments with strict egress controls — the default executor behavior of dynamically downloading Python dependencies via uv or pip will fail. Operators need a fully offline strategy where all ingestion plugin dependencies are resolved and bundled at image build time, with zero outbound calls to public package indexes during execution. DataHub v1.0.2 introduced native support for this use case through pre-built bundled virtual environments and a hardened -locked image variant, replacing earlier workarounds that intercepted uv subprocess calls.

Error Messages

ERROR: Could not find a version that satisfies the requirement acryl-datahub[...] (from versions: none)
WARNING: Retrying (Retry(total=N, ...)) after connection broken by 'NewConnectionError'
pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded
uv: error: Network connectivity required but not available

You Might Be Asking

Does DataHub v1.0.2 still support offline dependency resolution for the Remote Ingestion Executor?
What changed in v1.0.2 that might break my existing uv wrapper / --find-links interception approach?
What is the -locked image variant and when should I use it?
How do I build a bundled RIE image that does not require any PyPI access at runtime?
How do I tell DataHub to use the pre-built bundled venv instead of downloading a fresh CLI version?
Will ingestion runs show as "Failed" in the UI even when metadata was ingested successfully?
What happens to extra pip dependencies specified in the DataHub UI in an offline setup?
How does CLI version backward compatibility work between DataHub Cloud and an older RIE?

Solution

Overview of the Native Offline Solution in v1.0.2

DataHub v1.0.2 ships two complementary mechanisms for fully offline ingestion execution. Both are built around pre-building isolated virtual environments per ingestion plugin at image build time rather than at runtime.

Bundled Virtual Environments (all image variants): The executor pre-builds isolated virtualenvs for all standard ingestion plugins under /opt/datahub/venvs/<plugin>-bundled/ at image build time. When a matching bundled venv is found at runtime, no uv pip install or PyPI call is made.
The -locked Image Variant (recommended for air-gapped environments): The datahub-executor:<version>-locked image strips the pip and uv binaries from the final image entirely and locks all index URLs to an unreachable loopback address (http://127.0.0.1:1/simple), making outbound PyPI calls impossible by design. It ships with all bundled venvs pre-installed and is purpose-built for private subnet or air-gapped deployments.

Important Behavioral Change in v1.0.2 Affecting Prior Workarounds

Prior to v1.0.2, some operators intercepted uv subprocess calls (e.g., by replacing the uv binary with a shell wrapper that injected --find-links /path/to/local/wheels). In v1.0.2, the executor's venv creation process was changed: pip download is now invoked first to fetch a constraints file for acryl-datahub, and then uv is called separately for installation. This two-step process means a uv-only binary wrapper will not intercept the initial pip download step, causing failures in offline environments even when the wrapper is in place. You must also intercept the pip binary (or switch to the native bundled venv approach) to fully prevent outbound network calls.

Recommended Approach: Build a Bundled RIE Image

The recommended path for air-gapped environments is to pre-build venvs for each required ingestion source at image build time using DataHub's bundling helper scripts. The general pattern is shown below. Adjust the list of sources to match your environment.

# Example Dockerfile — bundled offline RIE image
# Base image: use the appropriate DataHub executor base for your version
FROM datahub-executor:<version>-slim AS builder

# Copy the bundling helper scripts provided by DataHub engineering
COPY build_bundled_venvs_unified.py /opt/datahub/scripts/
COPY requirements-sources.txt /opt/datahub/scripts/

# Set index URLs to your internal mirror (or a private index) at build time.
# Leave empty or point to an internal PyPI mirror if you have one.
ARG INTERNAL_PIP_INDEX=https://<your-internal-pypi-mirror>/simple
ENV PIP_INDEX_URL=${INTERNAL_PIP_INDEX}
ENV UV_INDEX_URL=${INTERNAL_PIP_INDEX}

# Build bundled venvs for each ingestion plugin
# The script creates isolated venvs under /opt/datahub/venvs/<plugin>-bundled/
RUN python /opt/datahub/scripts/build_bundled_venvs_unified.py \
    --sources glue,athena,redshift,postgres,mysql,kafka,dbt,looker,lookml,tableau,clickhouse \
    --output-dir /opt/datahub/venvs

# ---------- Final image ----------
FROM datahub-executor:<version>-slim

# Copy pre-built venvs from the builder stage
COPY --from=builder /opt/datahub/venvs /opt/datahub/venvs

# Block all runtime PyPI access by pointing indexes to an unreachable address.
# This is already done for you in the -locked variant; add it manually for slim.
ENV UV_INDEX_URL=http://127.0.0.1:1/simple
ENV UV_DEFAULT_INDEX=http://127.0.0.1:1/simple
ENV UV_EXTRA_INDEX_URL=""
ENV PIP_INDEX_URL=http://127.0.0.1:1/simple
ENV PIP_EXTRA_INDEX_URL=""

If you also need to bundle private or internal Python packages (e.g., internal libraries not available on any public index), copy the pre-downloaded wheel files into the image and install them into the relevant bundled venvs during the builder stage before copying to the final image.

# Example: Installing internal wheels into a bundled venv during the builder stage
COPY ./internal-wheels/ /tmp/internal-wheels/

RUN /opt/datahub/venvs/<plugin>-bundled/bin/pip install \
    --no-index \
    --find-links /tmp/internal-wheels/ \
    your-internal-package==<version>

Step 1 — Set the CLI Version to `bundled` in the DataHub UI

After deploying the new image, you must configure each ingestion source to use the pre-built bundled venv rather than attempting to resolve a specific CLI version from PyPI at runtime.

In the DataHub UI, navigate to Ingestion → select the ingestion source.
Open Advanced Settings.
Set the CLI Version field to the string bundled.
Save and re-run the ingestion source.
Repeat for every ingestion source in your environment.

Warning: If you do not set the CLI version to bundled after upgrading, the DataHub backend will default to a newer CLI version string. The executor will attempt to download that version from PyPI at runtime, which will fail in an offline environment.

Step 2 — Verify Ingestion Execution Logs

After setting the CLI version to bundled and redeploying your image, trigger a test ingestion run and confirm in the executor logs that no outbound pip or uv network calls are made.

# Expected log output when a bundled venv is used successfully
INFO  Found bundled venv for plugin 'glue' at /opt/datahub/venvs/glue-bundled
INFO  Skipping dependency installation — using pre-built venv
INFO  Starting ingestion with bundled environment

Handling Extra Pip Dependencies Specified from the UI

Any extra pip dependencies added through the DataHub UI's ingestion source configuration cannot be resolved dynamically in an offline environment. If additional packages are required, they must be pre-installed into the relevant bundled venv at image build time. Runtime UI-specified extra dependencies will be ignored or cause failures when PyPI is unreachable.

CLI Version Backward Compatibility

When using bundled venvs, the executor behaves as if an older CLI version is hardcoded in the recipe. In general, running an older RIE (with bundled venvs) against a newer DataHub Cloud backend is supported from an ingestion correctness perspective, but the DataHub team recommends keeping the RIE version as close to the server version as possible to avoid subtle compatibility gaps over time.

Choosing the Right Image Variant

All plugins are standard (e.g., Glue, Athena, Redshift, Postgres, MySQL, Kafka, dbt, Looker, LookML, Tableau): Use datahub-executor:<version>-locked directly. No wrapper needed; outbound PyPI calls are structurally impossible.
Custom or non-standard plugins are required: Extend datahub-executor:<version>-slim and pre-build venvs for your custom plugins using the bundling scripts.
Hybrid (standard plugins + occasional custom extras or internal packages): Extend datahub-executor:<version>-slim, pre-build all required venvs including internal packages, and set index environment variables to block runtime resolution.

Additional Notes

Known Issue — Ingestion Runs Shown as "Failed" When Using Bundled Venvs (v1.0.2 and v1.0.3): When a bundled venv is used, ingestion runs may be marked as Failed in the DataHub UI even when the ingestion itself completed successfully and all metadata was written correctly. The failure occurs in the post-run cleanup step, not during data ingestion. This is a known issue being addressed; a fix is included in v2.0.1 and later. In the interim, verify actual ingestion success by inspecting the execution logs directly rather than relying solely on the UI status indicator. Evaluate whether this false-failure status will trigger alerting (e.g., PagerDuty) in your environment, and configure alert suppression or filtering as appropriate until the fix is available.

Cloud-Specific Documentation: The bundled ingestion venv documentation at docs.datahub.com/docs/docker/bundled-ingestion-venvs was initially written targeting OSS deployments. Cloud-specific step-by-step guidance and officially supported example scripts for DataHub Cloud are being developed. Contact DataHub support for the latest example Dockerfile and bundling scripts if you are deploying on DataHub Cloud.

Runtime Index Environment Variables (Unlocked Variants): On slim and full image variants, the executor respects UV_INDEX_URL, UV_DEFAULT_INDEX, and UV_EXTRA_INDEX_URL at runtime, allowing configuration of an internal PyPI mirror without binary-level interception. This is an alternative to the -locked image for environments that have an internal mirror but not full air-gap.

Version Applicability: The bundled venv feature and -locked image variant were introduced and hardened in DataHub v1.0.2. The behavior described in this article has been tested against v1.0.2 and v1.0.3. Verify bundled plugin coverage for your specific version before removing any existing interception mechanisms.

Support Boundary: Using the bundled venv approach with custom internal packages is functional but extends beyond the standard tested configuration. Coordinate with your DataHub account team to ensure your customized image is reviewed and to plan for ongoing maintenance as the executor evolves.