AWS Glue Catalog Integration Issues

AWS Glue Data Catalog metadata is not syncing correctly to DataHub, or lineage from Glue ETL jobs is missing. This commonly occurs due to IAM permission issues, incorrect region configuration, or missing Glue job metadata.

Common Error Messages:

• AccessDeniedException: User is not authorized to perform: glue:GetDatabases
• glue:GetTable access denied
• Invalid AWS region specified

You Might Be Asking:

• What IAM permissions does DataHub need for Glue?
• How do I extract lineage from Glue ETL jobs?
• Can DataHub sync Glue crawlers?

1. Configure Glue source with proper IAM:

Example for AWS Glue. This can be adapted for other cloud data catalog services.

source:
  type: glue
  config:
    aws_region: "us-east-1"

    # Use IAM role or access keys
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    # Or use IAM role (recommended for EKS/EC2)

    # Extract options
    extract_transforms: true  # Include Glue ETL jobs
    extract_owners: true

    # Filter databases
    database_pattern:
      allow: ["analytics.*", "warehouse.*"]
      deny: ["temp.*"]

2. Required IAM permissions:

Example for AWS Glue. This can be adapted for other cloud data catalog services.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabases",
        "glue:GetDatabase",
        "glue:GetTables",
        "glue:GetTable",
        "glue:GetPartitions",
        "glue:GetJobs",
        "glue:GetJob",
        "glue:GetCrawlers",
        "glue:GetCrawler"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::your-bucket/*"
    }
  ]
}

3. Extract Glue ETL job lineage:

Example for AWS Glue. This can be adapted for other cloud data catalog services.

source:
  type: glue
  config:
    extract_transforms: true

    # This will create lineage from:
    # - Glue job source tables → target tables
    # - Based on job properties and script analysis

    # Filter specific jobs
    job_name_pattern:
      allow: ["prod.*", "analytics.*"]

4. Handle Glue tables with S3 locations:

Example for AWS Glue. This can be adapted for other cloud data catalog services.

source:
  type: glue
  config:
    # Include S3 path information
    extract_s3_lineage: true

    # This creates relationships:
    # S3 location → Glue table
    use_s3_bucket_tags: true
    use_s3_object_tags: false

5. For Lake Formation managed tables:

Example for AWS Glue with Lake Formation. This can be adapted for other cloud data catalog services.

source:
  type: glue
  config:
    # Additional Lake Formation permissions needed
    # in IAM policy:
    # - lakeformation:GetDataAccess
    # - lakeformation:GrantPermissions

    extract_lake_formation_permissions: true

Glue Catalog metadata includes table schema, partitions, and S3 locations. For complete lineage, combine Glue ingestion with S3, Athena, and EMR ingestion. Glue crawler metadata can be used to track schema evolution.

• Glue Source
• AWS IAM Configuration