Dremio Connector Configuration and URN Issues

Dremio connector ingestion fails when run from CLI, or there are issues defining correct URNs for Dremio entities, especially when Dremio connects to external data sources with different naming conventions.

Note: For DataHub Cloud customers, infrastructure-level changes are managed by DataHub Support.

1. Configure Dremio connector:

source:
  type: dremio
  config:
    username: "${DREMIO_USER}"
    password: "${DREMIO_PASSWORD}"
    host_port: "dremio.company.com:9047"
    # For external sources through Dremio
    platform_instance: "prod"

2. Understand Dremio URN structure:

   • Format: urn:li:dataset:(urn:li:dataPlatform:dremio,source.schema.table,PROD)
   • Dremio sources (S3, HDFS, etc.) need proper naming
   • External sources may need manual URN mapping

3. Configure for external sources:

   • When Dremio connects to S3, Snowflake, etc., URNs should reflect source platform
   • Use platform_alias to map Dremio tables to actual source platforms
   • Consider ingesting directly from source platforms for clearer lineage

4. Troubleshoot CLI ingestion:

   • Verify Dremio API accessibility from CLI environment
   • Check authentication credentials
   • Review Dremio permissions for metadata access
   • Test connection with Dremio ODBC/JDBC first

Test configurations in a development environment before applying to production. Always backup configurations before making changes.

• DataHub Documentation
• Dremio Source

• 5184, 4311