S3 Folder-Level Ingestion Failures

After updating S3 ingestion configurations to create dataset entities at the folder level (rather than individual files), ingestions consistently failed with an "Aborted" status. This prevented users from organizing S3-based data assets in a hierarchical folder structure within DataHub.

Common Error Messages:

• Ingestion status: Aborted
• S3 folder traversal failed
• Timeout during S3 listing operations

The S3 connector has been fixed to properly handle folder-level entity creation without timing out or aborting. Ensure your configuration uses the correct path specifications:

Example for S3. This can be adapted for other data lake sources.

source:
  type: s3-data-lake
  config:
    path_specs:
      - include: "s3://bucket-name/folder-path/**/*"
        # Create entities at folder level
        table_name: "{folder}"

      # Or specify exact folder patterns
      - include: "s3://bucket-name/data/{table}/**/*"
        table_name: "{table}"

    aws_config:
      aws_region: "us-east-1"
      # Ensure proper credentials

Fixed in version v0.3.12. The fix improves S3 listing performance and timeout handling for large folder structures. If you have very large S3 buckets, consider using more specific path patterns to reduce scanning time.

• S3 Data Lake Source
• Path Specifications