Redshift Connector Issues and Phantom Databases

Redshift ingestion creates unexpected 'phantom' databases in DataHub that don't exist in the actual Redshift instance, or connection failures occur when attempting to connect to Redshift for metadata extraction. Additionally, manually created lineage to Redshift assets may fail to persist or display correctly.

Note: For DataHub Cloud customers, infrastructure-level changes are managed by DataHub Support.

1. Review Redshift connection configuration:

source:
  type: redshift
  config:
    host_port: "your-cluster.region.redshift.amazonaws.com:5439"
    database: "your_database"
    username: "${REDSHIFT_USER}"
    password: "${REDSHIFT_PASSWORD}"
    # Limit which databases are scanned
    database_pattern:
      allow: ["prod_db", "analytics_db"]
      deny: ["test_.*", "temp_.*"]

2. Check for cross-database visibility:

   • Phantom databases may appear if the Redshift user has access to see metadata from databases they shouldn't
   • Verify the service account only has USAGE rights on intended databases
   • Use database_pattern to explicitly include/exclude databases

3. Troubleshoot connection failures:

   • Verify security group rules allow inbound connections from DataHub
   • Confirm the Redshift cluster is publicly accessible if ingesting from outside VPC
   • Check that SSL requirements match between connector config and cluster settings
   • Test connection using psql or another Redshift client from the same network

4. For manual lineage issues:

   • Ensure Redshift URNs follow the correct format: urn:li:dataset:(urn:li:dataPlatform:redshift,database.schema.table,PROD)
   • Verify the target Redshift dataset exists in DataHub before creating lineage
   • Check if lineage was actually persisted using the GraphQL API

Test configurations in a development environment before applying to production. Always backup configurations before making changes.

• DataHub Documentation
• Redshift Source

• 5763, 5716, 5047, 5628