S3 Connector Metadata and Lineage Issues

S3 connector fails to extract proper metadata for S3 assets, or lineage tracking for file-level operations is incomplete or missing, preventing proper data lineage visualization for S3-based data flows.

Note: For DataHub Cloud customers, infrastructure-level changes are managed by DataHub Support.

1. Configure S3 metadata extraction:

source:
  type: s3
  config:
    aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
    aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    aws_region: "us-east-1"
    path_specs:
      - include: "s3://your-bucket/data/**/*.parquet"
        table_name: "your_table_name"
    # Enable detailed metadata
    profile_patterns:
      allow: [".*"]

2. Extract file-level lineage:
   • S3 connector supports basic lineage for file-based workflows
   • For complex lineage, use custom emitters to create lineage:

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import UpstreamLineageClass, UpstreamClass

emitter = DatahubRestEmitter("https://your-datahub.acryl.io")

upstream_lineage = UpstreamLineageClass(
    upstreams=[
        UpstreamClass(
            dataset=make_dataset_urn("s3", "your-bucket/source/file.parquet"),
            type="COPY"
        )
    ]
)

# Emit for target dataset

3. Troubleshoot S3 connection:
   • Verify IAM credentials have S3 read permissions
   • Check bucket policies allow access from the service account
   • Ensure bucket region matches the configured region

Test configurations in a development environment before applying to production. Always backup configurations before making changes.

• DataHub Documentation

• 5721