Ingestion Pipeline Fails with Authentication Errors

Ingestion pipelines fail to connect to data sources due to authentication failures. This commonly occurs when credentials expire, are incorrectly configured, or when the service account lacks necessary permissions to access the source system.

Common Error Messages:

• Authentication failed for user
• Access denied: insufficient permissions
• 401 Unauthorized
• 403 Forbidden
• PermissionError: Unable to access resource

You Might Be Asking:

• Why is my ingestion pipeline suddenly failing with authentication errors?
• How do I update credentials for my data source connection?
• What permissions does the service account need?

1. Verify your credentials are correct and not expired:

This is only applicable to CLI-based ingestions, not generally used in DataHub Cloud/SaaS

# Check your recipe configuration
datahub check plugins

2. Update credentials in your ingestion recipe:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    username: "${SNOWFLAKE_USER}"
    password: "${SNOWFLAKE_PASSWORD}"
    # Or use key-pair authentication
    private_key_path: "/path/to/private_key.p8"
    private_key_password: "${KEY_PASSWORD}"

3. For managed ingestion, update credentials through the UI:

   • Navigate to Ingestion → Sources
   • Select your source and click "Edit"
   • Update authentication credentials
   • Test the connection before saving

4. Verify service account permissions in the source system (for Snowflake example):

Example for Snowflake. This can be adapted for other data sources/connectors.

GRANT USAGE ON WAREHOUSE your_warehouse TO ROLE datahub_role;
GRANT USAGE ON DATABASE your_database TO ROLE datahub_role;
GRANT USAGE ON ALL SCHEMAS IN DATABASE your_database TO ROLE datahub_role;
GRANT SELECT ON ALL TABLES IN DATABASE your_database TO ROLE datahub_role;

When using environment variables for credentials, ensure they are properly set in the environment where the ingestion job runs. For Kubernetes deployments, use secrets rather than configmaps.

• Ingestion Sources Overview
• Authentication Guide