S3 Data Lake Metadata Ingestion

Metadata from S3-based data lakes is not being ingested correctly, or file-level metadata and schema information is missing. This typically occurs when S3 paths are not structured, schema files are missing, or IAM permissions are insufficient.

You Might Be Asking:

• How do I catalog S3 data in DataHub?
• Can DataHub extract schema from Parquet/Avro files?
• How do I organize S3 paths as datasets?

1. Configure S3 data lake source:

Example for AWS S3. This can be adapted for other object storage systems like Azure Blob Storage or Google Cloud Storage.

source:
  type: s3
  config:
    aws_region: "us-east-1"
    # Credentials
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

    # S3 path patterns
    path_specs:
      - include: "s3://my-bucket/data/analytics/**/*.parquet"
        exclude:
          - "**/temp/**"
          - "**/_SUCCESS"

    # Schema extraction
    use_s3_bucket_tags: true
    use_s3_object_tags: true

    # Table naming from S3 paths
    table_pattern: "s3://my-bucket/data/{table}/**"

2. Extract schema from Parquet files:

Example for AWS S3 with Parquet files. This can be adapted for other file formats like Avro or ORC.

source:
  type: s3
  config:
    path_specs:
      - include: "s3://bucket/analytics/**/*.parquet"

    # Parquet schema extraction
    extract_schema: true

    # Sample files for schema
    sample_files: 5

    # Define table structure
    table_pattern: "s3://bucket/{database}/{table}/**"

3. Use AWS Glue Catalog for S3 schema:

Example for AWS S3 with Glue Catalog. This can be adapted for other catalog systems.

# Better approach: Combine S3 with Glue
source:
  type: glue
  config:
    aws_region: "us-east-1"
    # Glue provides schema for S3 tables
    extract_s3_lineage: true

    # This creates datasets with S3 locations
    # and proper schema information

4. Required IAM permissions for S3:

Example for AWS S3. This can be adapted for other cloud providers' object storage services.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetBucketTagging"
      ],
      "Resource": "arn:aws:s3:::my-bucket"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectTagging",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

5. Handle partitioned S3 data:

Example for AWS S3 with partitioned data. This can be adapted for other object storage systems.

source:
  type: s3
  config:
    # Path with partitions
    path_specs:
      - include: "s3://bucket/data/year=*/month=*/day=*/*.parquet"

    # Extract partition keys
    partition_pattern: "year={year}/month={month}/day={day}"

    # Group partitions into single dataset
    table_pattern: "s3://bucket/data/{table}/**"

S3 ingestion works best when combined with AWS Glue Catalog or when files follow consistent naming conventions. For schema evolution tracking, consider using Glue crawlers. S3 tags can be mapped to DataHub tags for governance.

• S3 Data Lake Source
• AWS Integration