Skip to main content

Resolving Private Network Access for Package Installation

Angelo Bresci
Updated

Area: Deployment Issues

Sub-Area: Remote Executor Network Configuration

Issue

Organizations using AWS PrivateLink or other private network configurations encounter package installation failures when running DataHub ingestion recipes. The Remote Ingestion Executor (RIE) fails to create virtual environments because it cannot access PyPI (pypi.org) to download required Python packages, resulting in blocked ingestion workflows in network-restricted environments.

Error Messages

  • error sending request for url (https://pypi.org/simple/acryl-datahub/)
  • Unable to reach PyPI for package installation

You Might Be Asking

  • Can PyPI be added to Private DNS resolution for PrivateLink access?
  • How can I run ingestion in a network-restricted environment?
  • Why does the Remote Executor need internet access?
  • Can I disable dynamic virtual environment creation?

Solution

The Remote Executor requires access to a Python package index to install dependencies during virtual environment creation. Here are the recommended solutions for private network environments:

Option 1: Configure Internal PyPI Mirror (Recommended)

  1. Set up an internal PyPI mirror using tools like Artifactory, Nexus, or AWS CodeArtifact
  2. Configure your Remote Executor to use the internal mirror by setting environment variables: 
    PIP_INDEX_URL=https://your-internal-mirror/simple
PIP_EXTRA_INDEX_URL=https://your-backup-mirror/simple
  3. Update your Remote Executor deployment configuration to include these environment variables
  4. Restart the Remote Executor to apply the new configuration

Option 2: Use Pre-Bundled Executor Images

  1. Create a custom Docker image extending the DataHub Remote Executor base image
  2. Pre-install required plugins in your custom image: 
    FROM acryldata/datahub-ingestion-slim:latest
RUN pip install --upgrade acryl-datahub[]
  3. Build and deploy the custom image to your container registry
  4. Update your Remote Executor deployment to use the custom image
  5. Configure ingestion recipes to use bundled virtual environments (requires DataHub 0.3.17+)

Option 3: Network-Level PyPI Access

  1. Configure your network infrastructure to allow outbound HTTPS access to pypi.org
  2. Ensure the Remote Executor's network security groups allow outbound connections on port 443
  3. Add firewall rules to permit access to PyPI's IP ranges

Configuration for Bundled Virtual Environments (DataHub 0.3.17+)

For organizations requiring fully air-gapped deployments:

  1. Upgrade your DataHub instance to version 0.3.17 or higher
  2. Create a Dockerfile with pre-installed dependencies: 
    FROM acryldata/datahub-ingestion:0.3.17
RUN pip install --upgrade acryl-datahub[glue,snowflake,s3]
# Add other required plugins as needed
  3. Build and tag the image: 
    docker build -t /datahub-executor:bundled .
  4. Deploy the custom image to your Remote Executor infrastructure
  5. Update ingestion recipes to reference the bundled virtual environment configuration

Additional Notes

AWS PrivateLink is designed for private connectivity to specific services and cannot proxy arbitrary public internet destinations like PyPI. The bundled virtual environment feature requires DataHub 0.3.17+ and may have limitations on CLI version flexibility. Pre-bundled images will require manual updates when new plugin versions are released. Consider the trade-offs between security isolation and operational flexibility when choosing your approach.

Related Documentation

Tags: private-network, privatelink, remote-executor, pypi, package-installation, air-gapped, network-security, ingestion-setup, deployment-configuration, virtual-environment

Related articles