Authentication Errors When Running DataHub CLI Ingestion with Service Account Tokens – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: CLI Authentication / datahub-rest Sink Configuration

Issue

When running DataHub CLI ingestion from an external environment (such as a Kubernetes cluster) using the datahub-rest sink pointed at a DataHub Cloud instance, users may encounter an authentication error even when the same recipe appears to work in other contexts. The error is a 401 Unauthorized response returned by the DataHub Cloud GMS /config endpoint during the initial connectivity check performed by the REST emitter. Root causes include expired or revoked Personal Access Tokens (PATs), YAML formatting errors that cause the token to be parsed incorrectly, token/environment mismatches (e.g., using a token generated in one DataHub Cloud environment against a different environment's GMS endpoint), and ingestion scripts that fail to read or inject the token value correctly at runtime.

Error Messages

datahub.configuration.common.ConfigurationError: Unable to connect to https://<your-instance>.acryl.io/gms/config - got an authentication error: Unauthorized.

You Might Be Asking

Why does my ingestion recipe work in the DataHub UI but fail when run from a Kubernetes cluster or automated script?
Why does my service account token return a 401 error even though it was recently created?
Can I reuse a token from one DataHub Cloud environment (e.g., staging) against a different environment (e.g., production)?
How do I confirm whether the problem is a bad token or a misconfigured recipe?

Solution

Work through the following checks in order. Each one addresses a distinct, commonly observed root cause for this error.

Fix YAML formatting in the recipe.
Malformed YAML indentation or missing whitespace between the key and value can cause the token field to be parsed as null, resulting in requests being sent with no Authorization header. Verify your sink block matches the structure below exactly, with consistent 4-space indentation and a space after the colon:
```
sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.acryl.io/gms"
    token: "<your-personal-access-token>"
```
Common mistakes to avoid:
- token:"value" — missing space after the colon.
- Inconsistent indentation between server: and token: lines.
- Mixing single quotes and double quotes in ways that interfere with token string parsing.
Regenerate or verify the Personal Access Token.
PATs can expire or be revoked. If the token previously worked and now fails, generate a new one:
1. Log in to the DataHub Cloud UI for the target environment.
2. Navigate to Settings → Access Tokens.
3. Click Generate new token. Set an appropriate expiration (or select "No expiration" for long-running automation).
4. Copy the token value immediately — it is only shown once.
5. Update your recipe or secret store with the new token value.
Ensure the token matches the target environment.
DataHub Cloud environments (e.g., staging and production) are fully isolated. Each environment has its own token store and signing keys. A token generated in one environment will always return a 401 when used against a different environment's GMS endpoint. Confirm that:
- The token was generated by logging in to the same DataHub Cloud instance that the server field in your recipe points to.
- The server URL and the token origin URL share the same hostname (e.g., both use <your-instance>-stg.acryl.io for staging, or both use <your-instance>.acryl.io for production).
Example of a correct staging configuration:
```
sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>-stg.acryl.io/gms"
    token: "<token-generated-from-staging-ui>"
```
Verify that the token has sufficient privileges.
The user account or service account used to generate the token must have ingestion-level permissions. Tokens generated by read-only accounts or accounts lacking the MANAGE_INGESTION or equivalent privilege will be rejected. Confirm with your DataHub Cloud administrator that the account holds the necessary role before regenerating a token.
Check that your script or automation is reading the token correctly.
When injecting the token via an environment variable or secrets manager (e.g., Kubernetes Secrets), verify that the variable is correctly exported and non-empty at the time the ingestion process starts. A common pattern using an environment variable reference in YAML:
```
sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.acryl.io/gms"
    token: "${DATAHUB_TOKEN}"
```
To test whether the token is being read at all, temporarily print or log the first few characters of the resolved token value before the ingestion command runs. If the value is empty or None, the issue is in how the secret is mounted or exported — not in DataHub itself.

Example diagnostic shell snippet:
```
#!/bin/bash
# Confirm the token variable is populated before running ingestion
if [ -z "$DATAHUB_TOKEN" ]; then
  echo "ERROR: DATAHUB_TOKEN is not set or is empty."
  exit 1
fi
echo "Token loaded (first 6 chars): ${DATAHUB_TOKEN:0:6}..."
datahub ingest -c /path/to/recipe.yml
```
Test connectivity independently of the full pipeline.
Use the DataHub CLI to validate the connection and token before running a full ingestion:
```
datahub check server-config \
  --server "https://<your-instance>.acryl.io/gms" \
  --token "<your-personal-access-token>"
```
A successful response confirms authentication is working. A 401 at this stage confirms the token or server URL is incorrect — resolve this before proceeding with ingestion.

Additional Notes

The ConfigurationError: ... got an authentication error: Unauthorized message is generated by the DataHub REST emitter (rest_emitter.py) when it receives an HTTP 401 response from the GMS /config endpoint during startup. This is intentional behavior — it is not a product bug. The GMS is correctly rejecting an invalid, missing, or mismatched token. This error pattern does not indicate a regression in the DataHub product; it is always caused by a token or configuration problem on the client side. Ingestion recipes that appear to work in the DataHub Cloud UI use session-based authentication managed by the browser, which is separate from the Bearer token authentication used by the CLI and SDK — so a working UI experience does not guarantee a CLI token is valid.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation