Ingestion Recipe Secret Management

Storing credentials and secrets in ingestion recipes poses security risks. Organizations need secure methods to manage sensitive configuration data without exposing credentials in plain text.

You Might Be Asking:

• How do I securely store ingestion credentials?
• Can I use environment variables in recipes?
• What secret management systems are supported?

1. Use environment variables in recipes:

Example for Snowflake. This approach can be adapted for any data source/connector.

source:
  type: snowflake
  config:
    host_port: "account.snowflakecomputing.com"
    database: "analytics"
    # Reference environment variables
    username: "${SNOWFLAKE_USER}"
    password: "${SNOWFLAKE_PASSWORD}"
    warehouse: "${SNOWFLAKE_WAREHOUSE}"

2. Use AWS Secrets Manager:

Example for AWS Secrets Manager with Snowflake. This can be adapted for other cloud secret management services like Azure Key Vault or GCP Secret Manager.

source:
  type: snowflake
  config:
    host_port: "account.snowflakecomputing.com"
    # Reference AWS Secrets Manager
    username: "${AWS_SM:snowflake/datahub/username}"
    password: "${AWS_SM:snowflake/datahub/password}"

    # Configure AWS Secrets Manager access
    aws_region: "us-east-1"

3. Use Kubernetes secrets:

# Create Kubernetes secret
kubectl create secret generic datahub-ingestion-secrets \
  --from-literal=snowflake-user='datahub_user' \
  --from-literal=snowflake-password='secure_password' \
  -n datahub

# Reference in deployment
apiVersion: v1
kind: Pod
metadata:
  name: datahub-ingestion
spec:
  containers:
  - name: ingestion
    env:
    - name: SNOWFLAKE_USER
      valueFrom:
        secretKeyRef:
          name: datahub-ingestion-secrets
          key: snowflake-user
    - name: SNOWFLAKE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: datahub-ingestion-secrets
          key: snowflake-password

4. Use HashiCorp Vault:

Example for HashiCorp Vault with Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    host_port: "account.snowflakecomputing.com"
    # Reference Vault secrets
    username: "${VAULT:secret/data/snowflake#username}"
    password: "${VAULT:secret/data/snowflake#password}"

# Configure Vault access
vault:
  enabled: true
  address: "https://vault.company.com"
  token: "${VAULT_TOKEN}"
  # Or use Kubernetes auth
  auth_method: "kubernetes"

5. Encrypt recipe files:

# Use git-crypt or similar for recipe encryption
# Install git-crypt
apt-get install git-crypt

# Initialize in repository
cd datahub-recipes
git-crypt init

# Configure .gitattributes
echo "*.yml filter=git-crypt diff=git-crypt" >> .gitattributes
echo "*.yaml filter=git-crypt diff=git-crypt" >> .gitattributes

# Add collaborators
git-crypt add-gpg-user user@example.com

# Files are now encrypted in git

6. Best practices for secret management:

# Never commit secrets to version control
# Add to .gitignore:
*.env
*secrets*.yml
credentials.yml

# Use different secrets per environment
# dev-secrets.yml
# staging-secrets.yml  
# prod-secrets.yml

# Rotate secrets regularly
# Document rotation schedule
# Automate secret rotation where possible

# Principle of least privilege
# Grant only necessary permissions
# Use separate credentials per ingestion source
# Use read-only credentials when possible

For managed ingestion through DataHub UI, credentials are stored encrypted in the backend. Always use service accounts rather than personal credentials. Implement secret rotation policies. Audit secret access regularly.

• Secret Management
• Security Best Practices