Skip to main content

SQLGlot Upgrade Compatibility Issues

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: SQL Sources

Issue

After upgrading DataHub to newer versions (1.5.x), SQL-based ingestion sources like PostgreSQL and CockroachDB fail to initialize due to SQLGlot dependency version incompatibility. The ingestion sources become disabled during initialization, preventing the ingestion pipeline from running successfully.

Error Messages

  • Failed to find a registered source for type cockroachdb: cockroachdb is disabled due to an error in initialization
  • AttributeError: SQL_SECURITY
  • PipelineInitError: Failed to find a registered source for type <source_type>

You Might Be Asking

  • Why do my PostgreSQL and CockroachDB ingestions suddenly fail after upgrading DataHub?
  • How do I fix SQLGlot token-related errors in DataHub ingestion?
  • What version of SQLGlot should I use with DataHub 1.5.x?

Solution

This issue is caused by incompatible SQLGlot versions being installed alongside DataHub. The solution is to ensure you have the correct SQLGlot version that matches your DataHub installation.

  1. Clean Install SQLGlot (Recommended)

    Uninstall existing SQLGlot packages and reinstall DataHub with the sql-parser extra:

    pip uninstall -y sqlglot sqlglotrs sqlglotc
pip install --no-cache-dir --force-reinstall 'acryl-datahub[sql-parser]==<your-version>'
  2. Install with SQL Parser Extra

    When installing DataHub, include the sql-parser extra which automatically pulls compatible SQLGlot versions:

    pip install 'acryl-datahub[sql-parser,postgres,cockroachdb]'
  3. Verify Installation

    Check that only one SQLGlot package is installed:

    pip list | grep sqlglot

    You should see only sqlglot 30.0.3 (or the version compatible with your DataHub version) without any sqlglotrs or sqlglotc entries.

  4. Docker Environment Fix

    If using Docker, ensure your Dockerfile performs a clean installation:

    RUN pip uninstall -y sqlglot sqlglotrs sqlglotc && \
    pip install --no-cache-dir --force-reinstall \
    'acryl-datahub[postgres,cockroachdb]==1.5.0.19' \
    'acryl-datahub-cloud==1.0.1'
  5. Update Requirements Files

    Remove any explicit SQLGlot version pins from your requirements.txt or constraints files that might conflict with DataHub's requirements.

Additional Notes

This issue occurs because DataHub's SQL parsing functionality requires specific patches to SQLGlot that are only compatible with certain versions. Installing incompatible SQLGlot versions (especially newer releases like 30.x when DataHub expects 29.x, or vice versa) breaks the token handling and import mechanisms. Always use the sql-parser extra when installing DataHub to avoid version conflicts. For DataHub 1.5.x, ensure you're using compatible client versions - CLI version 1.5.0.19 with acryl-datahub-cloud 1.0.x for best compatibility.

Related Documentation

Tags: sqlglot, ingestion-failure, postgres, cockroachdb, upgrade, dependency-conflict, sql-parsing, version-compatibility, initialization-error