Remote Executor: Runtime Authentication for Private PyPI Repositories (AWS CodeArtifact) – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Remote Executor — Private Package Index Authentication

Issue

When running a DataHub Remote Executor and publishing custom ingestion transformer wheels to a private PyPI-compatible repository such as AWS CodeArtifact, teams need a way to authenticate pip or uv at runtime so that packages declared in a recipe's extra_pip_packages field can be installed dynamically — without baking every wheel version into the executor container image. This challenge is complicated by short-lived CodeArtifact authorization tokens and differences between persistent-host deployments (EC2) and ephemeral container deployments (AWS Fargate).

You Might Be Asking

Can the Remote Executor install extra_pip_packages from a private PyPI index at runtime?
Where should AWS CodeArtifact token refresh happen so ingestion runs always have a valid token?
How do I avoid rebuilding the executor image every time I publish a new wheel version?
How do I handle CodeArtifact authentication on Fargate, where I cannot run a cron job on the host?
Is there a startup hook in the executor to install packages like keyring without maintaining a custom image?

Solution

Prerequisites and Image Constraints

Use a slim or unbundled executor image. Variants with names containing locked-wolfi, locked-distroless, or similar have runtime dependency resolution deliberately disabled and their index pinned to an address that does not work. Private index authentication will not function with those variants.
Do not disable dependency resolution. Ensure the environment variable INGESTION_DEPENDENCY_RESOLUTION_ENABLED is not set to false. Its default value is true; if it is explicitly set to false the executor uses only bundled packages and skips runtime installs entirely.
Do not pin recipes to the bundled version. If a recipe is pinned to the bundled source version, the executor uses the prebuilt virtual environment and skips the runtime install step. Ensure your recipe is not forcing bundled resolution.
Use UV_EXTRA_INDEX_URL, not UV_INDEX_URL, for your private repo. The executor uses uv for package installation. Setting UV_EXTRA_INDEX_URL to your CodeArtifact repo keeps public PyPI as the primary index (so public dependencies continue to resolve) while adding CodeArtifact as an extra source for private wheels. uv also reads the pip equivalents (PIP_INDEX_URL, PIP_EXTRA_INDEX_URL), but the UV_* variables are preferred.

Option A — Fargate: Keyring-Based Token Refresh (Recommended for Ephemeral Containers)

On Fargate there is no persistent host to run a cron job. Instead, use the keyrings.codeartifact keyring backend so that uv fetches a fresh CodeArtifact token via the task's IAM role on every install invocation. No cron job, no sidecar, and no static token in the environment is needed.

Install the keyring packages at executor startup using the boot-time hook. The executor reads DATAHUB_EXECUTOR_EXTRA_PACKAGES and runs uv pip install on those packages before the worker comes up. Because keyring and keyrings.codeartifact are on public PyPI, no authentication is required for this step.
```
DATAHUB_EXECUTOR_EXTRA_PACKAGES="keyring keyrings.codeartifact"
```
There is also a companion variable DATAHUB_EXECUTOR_EXTRA_PACKAGES_STRICT (default false) that causes the container to fail at startup if the install errors — set it to true if you want a hard failure instead of a silent skip.

Note: Packages installed via DATAHUB_EXECUTOR_EXTRA_PACKAGES go into the executor's own base environment, not into the per-run virtual environment created for each ingestion job. Do not put your custom transformer wheel here — it would not be importable at ingestion time. Keep the transformer wheel in extra_pip_packages in the recipe, which installs into the correct per-run virtual environment.
Add the following environment variables to your Fargate task definition. These can be added as Type: value entries in the task definition environment block, the same way other executor variables such as AWS_REGION and DATAHUB_EXECUTOR_MODE are set.
```
DATAHUB_EXECUTOR_EXTRA_PACKAGES="keyring keyrings.codeartifact"
UV_KEYRING_PROVIDER=subprocess
UV_EXTRA_INDEX_URL=https://aws@<domain>-<account-id>.d.codeartifact.<region>.amazonaws.com/pypi/<repo>/simple/
```
The UV_EXTRA_INDEX_URL value intentionally contains a username (aws) but no password. When uv encounters an index URL with a username and no password, it calls out to the keyring provider. The keyrings.codeartifact backend recognizes the CodeArtifact host pattern and fetches a fresh authorization token using the ambient AWS credentials — on Fargate, those are the task IAM role credentials. This means every ingestion run gets a fresh token automatically.
Ensure the task IAM role has all required permissions. GetAuthorizationToken alone is not sufficient for the full install flow. The task role needs:
- sts:GetServiceBearerToken
- codeartifact:GetAuthorizationToken
- codeartifact:GetRepositoryEndpoint
- codeartifact:ReadFromRepository — scoped to the target repository

Add your private wheel to the recipe's extra_pip_packages. With the above in place, the only change needed in each recipe is declaring the package:

source:
  type: <your-source-type>
  config:
    # ... source config ...

transformers:
  - type: <your-transformer-type>

extra_pip_packages:
  - your-custom-transformer-wheel==<version>

Verify keyring availability. After deploying, exec into the container and confirm that the keyring backend is reachable:
```
keyring --version
```

Option B — EC2 Persistent Host: netrc File + Cron Refresh

If the executor runs on a persistent EC2 host, the recommended approach is to decouple the CodeArtifact token from the process environment (which is frozen at startup) and instead write the token to a ~/.netrc file that uv reads fresh on each invocation.

Do not export the token as an environment variable directly. The executor process reads its environment once at startup. If the CodeArtifact token is set in the environment at launch it will expire during a long-running executor session and every subsequent install will fail until the process is restarted.
Set the index URL as an environment variable. Export UV_EXTRA_INDEX_URL without a password; credentials will be supplied via ~/.netrc.
```
export UV_EXTRA_INDEX_URL="https://<domain>-<account-id>.d.codeartifact.<region>.amazonaws.com/pypi/<repo>/simple/"
```
Write a ~/.netrc file for the user account the executor process runs as, with a machine entry matching the CodeArtifact hostname:
```
machine <domain>-<account-id>.d.codeartifact.<region>.amazonaws.com
  login aws
  password <CODEARTIFACT_AUTH_TOKEN>
```
If you prefer a non-default location, set the NETRC environment variable to the full path of the file.

Run a cron job on the EC2 host (under the same user/role) to refresh the token every few hours (CodeArtifact tokens expire after 12 hours by default, so refreshing every 6–8 hours is safe). Because uv reads the ~/.netrc file fresh on each install invocation, a rewrite of the file is picked up by the next ingestion run without restarting the executor process.

#!/bin/bash
# Example cron script: place in crontab to run every 6 hours
TOKEN=$(aws codeartifact get-authorization-token \
  --domain <domain> \
  --domain-owner <account-id> \
  --query authorizationToken \
  --output text)

cat > ~/.netrc <<EOF
machine <domain>-<account-id>.d.codeartifact.<region>.amazonaws.com
  login aws
  password ${TOKEN}
EOF
chmod 600 ~/.netrc

Example crontab entry:

0 */6 * * * /path/to/refresh-codeartifact-token.sh

Add your private wheel to the recipe's extra_pip_packages exactly as shown in Option A, Step 4.

Option C — Fargate Fallback: Sidecar Container with Shared Volume

If installing additional packages into the executor image or using the keyring approach is not acceptable, a Fargate-compatible alternative to the cron approach is to run a sidecar container in the same task definition that fetches the CodeArtifact token and writes a ~/.netrc file to a shared volume mounted by the executor container (with NETRC pointing to the shared path). The sidecar can loop and refresh the file periodically. This is more complex to configure than Option A and is generally only recommended if the keyring approach cannot be used.

Additional Notes

The DATAHUB_EXECUTOR_EXTRA_PACKAGES boot-time hook installs into the executor's own base environment. It is intentionally separate from the per-run virtual environment used by ingestion jobs. Custom transformer wheels must remain in the recipe's extra_pip_packages.
CodeArtifact authorization tokens have a default maximum lifetime of 12 hours. Plan refresh intervals accordingly.
The UV_KEYRING_PROVIDER=subprocess variable instructs uv to invoke the keyring CLI as a subprocess for credential lookup. This requires keyring to be installed and on $PATH within the executor environment — verify with keyring --version.
By default, uv queries the primary index (public PyPI) before the extra index. Use UV_EXTRA_INDEX_URL (not UV_INDEX_URL) for your private repo to preserve public PyPI resolution for all standard dependencies.
The INGESTION_DEPENDENCY_RESOLUTION_ENABLED flag defaults to true. You do not need to set it explicitly unless you are overriding a prior explicit false value.
These approaches apply to DataHub Cloud Remote Executor deployments (customer-managed). They are not applicable to the DataHub-managed executor.