SQLite Version Error After DataHub SDK Upgrade

After upgrading to Acryl SDK version 0.3.13.2-acryl, Airflow DAGs fail during ingestion for multiple data sources with the error "PipelineInitError: Failed to configure the source: SQLite version 3.24.0 or later is required." This occurs even when the `OVERRIDE_SQLITE_VERSION_REQ` environment variable is set to True, which previously resolved the same issue.

Common Error Messages:

• PipelineInitError: Failed to configure the source (kafka): SQLite version 3.24.0 or later is required
• SQLite version 3.24.0 or later is required

You Might Be Asking:

• Why does the SQLite error occur after upgrading the SDK?
• Why doesn't OVERRIDESQLITEVERSION_REQ work anymore?
• How do I fix SQLite version issues in Airflow?
• Can I use an older SQLite version?

The SQLite version requirement error indicates that your Python environment has an outdated SQLite library. The SDK upgrade may have stricter version checks or changed dependencies.

Resolution Steps:

1. Upgrade SQLite in Your Environment:

For Airflow running in Docker:

FROM apache/airflow:2.7.0-python3.10

USER root
# Update SQLite to version 3.24.0+
RUN apt-get update && \
    apt-get install -y sqlite3 libsqlite3-dev && \
    apt-get clean

USER airflow
# Install DataHub SDK
RUN pip install acryl-datahub==0.3.13.2

For Airflow on Ubuntu/Debian:

# Update system packages
sudo apt-get update
sudo apt-get install -y sqlite3 libsqlite3-dev

# Rebuild Python if necessary
# (Python's sqlite3 module uses system SQLite)

2. Verify SQLite Version:

import sqlite3
print(f"SQLite version: {sqlite3.sqlite_version}")
# Should print 3.24.0 or higher

3. Environment Variable (Temporary Workaround):

While not recommended for production, you can bypass the check:

# In your Airflow DAG or environment
import os
os.environ['OVERRIDE_SQLITE_VERSION_REQ'] = 'True'

Why This Happens: - Python's sqlite3 module uses the system's SQLite library - Upgrading the DataHub SDK doesn't upgrade system SQLite - The SDK may have added new SQLite features requiring 3.24.0+ - Container images may have older base OS versions with outdated SQLite

• Upgrading SQLite is the proper long-term solution
• The OVERRIDE_SQLITE_VERSION_REQ flag may not work if the SDK genuinely requires newer SQLite features
• For managed Airflow services (MWAA, Cloud Composer), check provider documentation for SQLite version support
• This issue typically affects ingestion sources that use local metadata caching

• DataHub Airflow Integration
• CLI Installation

• 4959