Skip to main content

Airflow 3 Support in DataHub Connector

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Airflow Plugin Compatibility

Issue

Users attempting to upgrade their Apache Airflow environments to version 3.x find that the DataHub Airflow plugin does not function correctly. Airflow 3.0 introduced significant breaking changes — including reorganized import paths, removal of the v1 REST API, JWT-only authentication, listener hook signature changes, SubDAG removal, and a new SQL parser integration model — all of which require corresponding updates to the DataHub connector. Users blocked on this upgrade need to know whether Airflow 3.x is supported and what changes or versions are required.

Error Messages

  • ImportError: cannot import name 'BaseOperator' from 'airflow.models.baseoperator'
  • TypeError: __init__() got an unexpected keyword argument 'schedule_interval'
  • TypeError: __init__() got an unexpected keyword argument 'default_view'
  • RuntimeError: UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!
  • TypeError: cannot pickle '_thread.lock' object
  • 401 Unauthorized (when calling Airflow API with Basic Auth credentials)

You Might Be Asking

  • Does the DataHub Airflow plugin support Airflow 3.x?
  • Which version of DataHub introduced Airflow 3 support?
  • What do I need to change in my DAGs and configuration when upgrading to Airflow 3?
  • Does column-level lineage work with Airflow 3?
  • Can I still use the v1 DataHub Airflow plugin with Airflow 3?

Solution

  1. Upgrade to a supported DataHub version.

    Airflow 3.x support was added to the DataHub Airflow plugin starting with the following releases:

    • OSS DataHub: v1.4.0 and later
    • Managed DataHub (SaaS): v0.3.16 and later

    If you are running DataHub OSS v1.0.x, v1.1.x, or any version v1.4.0+, Airflow 3 support is available. Upgrade your acryl-datahub package accordingly:

    pip install "acryl-datahub[airflow]" --upgrade
  2. Use the v2 (listener-based) plugin only.

    The v1 DataHub Airflow plugin is not compatible with Airflow 3.x. Ensure you are using the v2 listener-based plugin. Verify your plugin configuration references the listener entrypoint and not the legacy operator-override approach.

  3. Update DAG definitions for Airflow 3.x compatibility.

    Replace deprecated DAG constructor parameters:

    # Airflow 2.x (deprecated) - DO NOT USE with Airflow 3
DAG("my_dag", schedule_interval="@daily", default_view="tree", ...)

# Airflow 3.x (required) - also works with Airflow 2.4+
DAG("my_dag", schedule="@daily", ...)
    • Replace schedule_interval= with schedule= (supported since Airflow 2.4).
    • Remove the default_view= parameter entirely — it was removed in Airflow 3.x.
  4. Replace SubDAGs with TaskGroups.

    SubDAGs were completely removed in Airflow 3.x. Migrate any SubDAG patterns to TaskGroups:

    # Old (Airflow 2.x) - SubDAG pattern
from airflow.operators.subdag import SubDagOperator

def subdag(parent_dag_name, child_dag_name, args):
    dag = DAG(f"{parent_dag_name}.{child_dag_name}", **args)
    # Add tasks...
    return dag

with DAG("parent_dag") as dag:
    subdag_task = SubDagOperator(
        task_id="subdag",
        subdag=subdag("parent_dag", "subdag", default_args)
    )

# New (Airflow 3.x) - TaskGroup pattern
from airflow.utils.task_group import TaskGroup

with DAG("parent_dag") as dag:
    with TaskGroup("task_group") as tg:
        task1 = BashOperator(...)
        task2 = BashOperator(...)

    Note: TaskGroup lineage is tracked at the individual task level, not as a separate DAG entity.

  5. Update Airflow environment configuration.

    Airflow 3.x moved several configuration keys. Update your environment variables:

    # Airflow 2.x configuration
AIRFLOW__WEBSERVER__WEB_SERVER_PORT=8080
AIRFLOW__WEBSERVER__BASE_URL=http://airflow.example.com
AIRFLOW__API__AUTH_BACKEND=airflow.api.auth.backend.basic_auth

# Airflow 3.x configuration
AIRFLOW__API__PORT=8080
AIRFLOW__API__BASE_URL=http://airflow.example.com
# AUTH_BACKEND no longer needed - uses SimpleAuthManager by default
  6. Update the kill switch mechanism (if used) for Airflow 3.x.

    Airflow 3.x prohibits database commits inside listener hooks (HA lock enforcement). If you use the DataHub plugin kill switch, switch from Airflow Variables to environment variables:

    # Airflow 2.x kill switch (Airflow Variable - NOT safe in Airflow 3.x)
# Variable.get("datahub_airflow_plugin_disable_listener", "false")

# Airflow 3.x kill switch (environment variable)
export AIRFLOW_VAR_DATAHUB_AIRFLOW_PLUGIN_DISABLE_LISTENER=true

    The DataHub plugin handles this detection automatically in supported versions.

  7. Understand the key architectural changes handled automatically by the plugin.

    The following changes are handled internally by the plugin and require no user action, but are useful to understand:

    • Import paths: Airflow 3.x reorganized modules (e.g., airflow.sdk.bases.operator). The plugin uses conditional imports with fallbacks automatically.
    • SQL lineage extraction: Airflow 3.x removed operator-specific extractors. The plugin now patches SQLParser.generate_openlineage_metadata_from_sql() to provide a unified lineage extraction point for all SQL operators. Column-level lineage is fully supported.
    • Listener hook signatures: Airflow 3.x removed the session parameter and added error to failure hooks. The plugin uses **kwargs for cross-version compatibility.
    • Template rendering: Airflow 3.x's RuntimeTaskInstance cannot be deep-copied. The plugin skips manual template rendering in Airflow 3.x since templates are already rendered before task execution.
    • OpenLineage provider: Airflow 3.x removed the legacy openlineage-airflow package. The plugin uses the native airflow.providers.openlineage provider and supplies its own SQL extractor implementation.
    • Threading: Threading is enabled by default and works correctly in Airflow 3.x. To disable it if needed: export DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD=false
  8. Verify the compatibility matrix for your setup.

    Supported Airflow versions with the DataHub v2 plugin:

    • Airflow 2.4.x – 2.9.x: Full support (v2 plugin)
    • Airflow 3.0.x and later: Full support (v2 plugin only)

Additional Notes

Airflow 3.x support was introduced in DataHub OSS v1.4.0 (released 2026-02-06) and DataHub SaaS v0.3.16 (released 2026-01-14). The v1 DataHub Airflow plugin (operator-override based) is not compatible with Airflow 3.x and will not receive Airflow 3 support. SubDAG lineage tracking is no longer available in Airflow 3.x because SubDAGs were completely removed from Airflow itself; migrate to TaskGroups instead. macOS users running Airflow 3.x integration tests locally may encounter SIGSEGV crashes caused by the setproctitle package after fork(); this is a macOS-specific issue unrelated to production Linux deployments. The REST API in Airflow 3.x is v2 only; v1 endpoints have been removed. API authentication now requires JWT tokens instead of HTTP Basic Auth.

Related Documentation

Tags: airflow, airflow-3, airflow-plugin, lineage, ingestion, upgrade, compatibility, sql-lineage, column-lineage, openlineage