AWS IAM Authentication for Data Sources (Aurora, MSK/Kafka)

Configuring DataHub ingestion to use AWS IAM authentication for data sources (such as Aurora PostgreSQL/MySQL or MSK Kafka) requires clarification, or IAM-based authentication is not working as expected.

Note: For DataHub Cloud customers, infrastructure-level changes are managed by DataHub Support.

1. Configure IAM authentication for Aurora:

This example is for Aurora PostgreSQL but can be adapted for Aurora MySQL or other AWS RDS databases supporting IAM authentication.

source:
  type: postgres  # or mysql
  config:
    host_port: "your-aurora-cluster.cluster-xxx.us-east-1.rds.amazonaws.com:5432"
    database: "your_database"
    # For IAM authentication
    use_iam_auth: true
    aws_region: "us-east-1"
    # Service account should have IAM role with rds:connect permission

2. Configure IAM authentication for MSK/Kafka:

This example is for AWS MSK (Managed Streaming for Apache Kafka) but can be adapted for other Kafka deployments with IAM authentication.

source:
  type: kafka
  config:
    connection:
      bootstrap: "boot-xxx.kafka.us-east-1.amazonaws.com:9098"
      schema_registry_url: "https://schema-registry.company.com"
    # MSK IAM authentication
    consumer_config:
      security.protocol: "SASL_SSL"
      sasl.mechanism: "AWS_MSK_IAM"
      sasl.jaas.config: "software.amazon.msk.auth.iam.IAMLoginModule required;"
      sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

3. Set up IAM permissions:

   • Executor service account needs IAM permissions
   • For Aurora: rds:connect on the database resource
   • For MSK: kafka-cluster:Connect, kafka-cluster:DescribeCluster, etc.
   • Attach IAM role to Remote Executor or provide credentials

4. Troubleshoot IAM auth:

   • Verify IAM policy is attached and active
   • Check AWS CloudTrail for authentication attempts and denials
   • Ensure database/cluster allows IAM authentication
   • Test IAM credentials manually with AWS CLI first

Test configurations in a development environment before applying to production. Always backup configurations before making changes.

• DataHub Documentation

• 5690, 4755