Issue
When upgrading from an earlier version of DataHub to DataHub 0.3.10 or later, users connecting to RedShift sources for ingestion have reported that some schemas that were previously ingested are suddenly missing.
Cause
DataHub introduced support for serverless tables in Redshift in version 0.3.10. In order to support this feature, the minimal permissions needed by the DataHub user account that connects to Redshift were changed.
The permissions required in versions 0.3.10 and LATER are documented here.
Administrators familiar with previous configuration requirements may notice that several, more granular GRANTs are needed in later versions.
Impact
If your Redshift connector was set up using the old permissions, ingestion will miss certain schemas.
Action Required:
New (Updated) Permission Configuration Prerequisites
(Again, for your reference)
Previous Configuration Prerequisites (DO NOT USE)
Remediation
- Applying the GRANT prerequisites from the "new (updated) permission" link above will resolve this issue permanently
Additional Notes
- DataHub's Redshift connector does support Redshift serverless schemas/instances as of version 0.3.10 and later (see our Documentation).
- Breaking changes affecting existing functionality will be clarified in DataHub Release Notes when introduced. Remediation steps will be documented as well.