Configuring AWS Secrets Manager with DataHub Remote Executor for Snowflake Connections – DataHub Customer Support Portal

```html

Area: Ingestion Issues

Sub-Area: Remote Executor / Secret Management

Issue

When deploying the DataHub Remote Executor on AWS and configuring it to resolve secrets from AWS Secrets Manager, ingestion jobs that reference secrets — including Snowflake private key credentials — may fail even when the secret naming conventions and environment variables appear to be correctly configured. A particularly subtle but critical constraint is that the BatchGetSecretValue IAM action does not support resource-level ARN scoping and must be granted against a wildcard (*) resource. Scoping BatchGetSecretValue to a specific ARN pattern (e.g., arn:aws:secretsmanager:<region>:<account-id>:secret:datahub-*) silently prevents the Remote Executor from resolving secrets at runtime, even though all other permissions appear valid.

Error Messages

datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (snowflake): Failed to connect to snowflake instance: Unable to load PEM file. See https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for more details. MalformedFraming

You Might Be Asking

Can I use AWS Secrets Manager instead of DataHub UI secrets with the Remote Executor?
Why is my Snowflake ingestion failing with a PEM/MalformedFraming error even though my secret is stored correctly in AWS Secrets Manager?
Why does my Remote Executor fail to resolve AWS secrets even though GetSecretValue is permitted on my secrets?
Does BatchGetSecretValue support resource-level scoping in IAM?
What IAM permissions does the Remote Executor task role need to read from AWS Secrets Manager?

Solution

Overview

AWS Secrets Manager is fully supported with the DataHub Remote Executor and is the recommended approach for DataHub Cloud customers — it keeps credentials inside your own environment and they are never stored in DataHub's systems. The configuration requires three components to work correctly: the right environment variables on the executor, a correctly named secret in AWS Secrets Manager, a properly formatted secret value, and a correctly scoped IAM policy.

Step 1: Enable AWS Secrets Manager on the Remote Executor

Set the following environment variables on the Remote Executor container or ECS task definition:

DATAHUB_EXECUTOR_AWS_SM_ENABLED=true
DATAHUB_EXECUTOR_AWS_SM_REGION=<your-aws-region>
# Optional: override the default prefix (default is "datahub-")
# DATAHUB_EXECUTOR_AWS_SM_PREFIX=datahub-

When using the DataHub Terraform module, add these under the environment block:

environment = [
  {
    name  = "DATAHUB_EXECUTOR_AWS_SM_ENABLED"
    value = "true"
  },
  {
    name  = "DATAHUB_EXECUTOR_AWS_SM_REGION"
    value = var.aws_region
  }
]

Step 2: Name Your Secrets Correctly in AWS Secrets Manager

The Remote Executor resolves recipe placeholders of the form ${SECRET_NAME} by looking up a secret named <prefix>SECRET_NAME in AWS Secrets Manager. With the default prefix of datahub-, a recipe reference of ${SNOWFLAKE_PRIVATE_KEY} will look up a secret named datahub-SNOWFLAKE_PRIVATE_KEY.

Step 3: Store the Secret Value as Plaintext

When creating a secret in the AWS Console, always use the Plaintext tab and paste only the raw value (e.g., the PEM contents of a private key). Do not use the "Key/value" (JSON) option for secrets referenced directly in recipe fields. Using the key/value option causes AWS to store the secret as a JSON object, and the entire JSON string gets substituted into the recipe field at runtime, causing parse failures.

To store a private key file as a plaintext secret via the AWS CLI:

aws secretsmanager put-secret-value \
  --secret-id datahub-SNOWFLAKE_PRIVATE_KEY \
  --secret-string file:///path/to/rsa_key.p8

To verify the stored value matches your original file:

aws secretsmanager get-secret-value \
  --secret-id datahub-SNOWFLAKE_PRIVATE_KEY \
  --query SecretString --output text > from_secrets_manager.pem

diff rsa_key.p8 from_secrets_manager.pem

Step 4: Configure the IAM Policy — Critical `BatchGetSecretValue` Requirement

The Remote Executor's ECS task IAM role must have the following Secrets Manager permissions. The most important constraint is that BatchGetSecretValue does not support resource-level ARN scoping — it must be granted with Resource: "*". Scoping it to a specific ARN pattern will silently prevent secret resolution even if all other permissions are correctly set.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGetAndDescribeSpecificSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:<your-region>:<your-account-id>:secret:datahub-*"
    },
    {
      "Sid": "AllowBatchGetSecretValueGlobal",
      "Effect": "Allow",
      "Action": "secretsmanager:BatchGetSecretValue",
      "Resource": "*"
    },
    {
      "Sid": "AllowListSecrets",
      "Effect": "Allow",
      "Action": "secretsmanager:ListSecrets",
      "Resource": "*"
    }
  ]
}

After updating the IAM policy, redeploy or restart the Remote Executor so it picks up the new permissions.

Step 5: Reference the Secret in Your Ingestion Recipe

Use the ${SECRET_NAME} placeholder syntax in your ingestion YAML. The executor resolves these at runtime from AWS Secrets Manager:

source:
  type: snowflake
  config:
    account_id: <your-snowflake-account-id>
    username: <your-snowflake-username>
    authentication_type: KEY_PAIR_AUTHENTICATOR
    private_key: '${SNOWFLAKE_PRIVATE_KEY}'
    role: <your-snowflake-role>
    warehouse: <your-snowflake-warehouse>
    include_tables: true
    include_views: true
    include_table_lineage: true
    stateful_ingestion:
      enabled: true

With the default prefix, ${SNOWFLAKE_PRIVATE_KEY} resolves to the AWS Secrets Manager secret named datahub-SNOWFLAKE_PRIVATE_KEY.

Step 6: Validate the Private Key (If PEM Errors Persist)

If you still see MalformedFraming or PEM parse errors after confirming the secret is stored as plaintext and the IAM policy is correct, validate the key itself outside of DataHub:

openssl pkey -in rsa_key.p8 -noout -text

If this command fails, regenerate the key pair:

# Generate a new PKCS#8 private key
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt

# Extract the public key to register on the Snowflake user
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub

Then register the new public key on your Snowflake user:

ALTER USER <your-snowflake-username> SET RSA_PUBLIC_KEY='<contents-of-rsa_key.pub>';

Additional Notes

The MalformedFraming PEM error is misleading — it occurs after the secret is successfully retrieved from AWS Secrets Manager, not during lookup. This means the error does not indicate a permissions or naming problem; rather, it points to the content of the secret value itself (e.g., JSON wrapping or a malformed key). If the same private key works when stored as a DataHub UI secret but fails via AWS Secrets Manager, the most productive debugging step is to diff the two values as shown in Step 3.

The BatchGetSecretValue IAM restriction (Resource: "*" required) is an AWS-imposed limitation, not a DataHub limitation. This behavior is documented in the AWS IAM actions reference for Secrets Manager. Failing to account for this is the most common reason AWS Secrets Manager integration appears configured correctly but does not function.

After any IAM policy change, the Remote Executor task must be restarted (e.g., by forcing a new ECS deployment) to pick up updated permissions. IAM policy changes are not reflected in running tasks until the task is restarted.