Airflow Plugin Version Compatibility After SDK Upgrade

After upgrading to DataHub SDK version 0.3.13.2-acryl, release notes indicate that the acryl-datahub-airflow-plugin has dropped support for Airflow versions less than 2.7. Users need clarification on whether this means older Airflow versions will continue to run but are unsupported, or if the plugin will completely fail to work with Airflow versions below 2.7.

You Might Be Asking:

• Will my Airflow 2.6 installation still work with the new plugin?
• Does "dropped support" mean it won't work at all or just isn't tested?
• What happens if I upgrade the plugin without upgrading Airflow?
• Should I upgrade Airflow before upgrading the DataHub plugin?

"Dropped support" for Airflow versions < 2.7 means the plugin may not function correctly or at all due to dependency changes and incompatibilities.

Understanding "Dropped Support":

When a library drops support for older versions, it typically means:

1. Code Compatibility: The plugin may use features only available in Airflow 2.7+
2. Dependencies: Required packages may conflict with older Airflow versions
3. No Testing: Older versions are not tested and may have unexpected behavior
4. No Bug Fixes: Issues on older Airflow versions won't be addressed

Compatibility Matrix:

| DataHub SDK Version | Minimum Airflow Version | Recommended Airflow Version |
| < 0.3.13.2          | 2.1.0                   | 2.6.0+                      |
| >= 0.3.13.2         | 2.7.0                   | 2.7.0+                      |
    

Recommended Upgrade Path:

Option 1: Upgrade Airflow (Recommended)

# Upgrade Airflow first
pip install --upgrade apache-airflow==2.7.3

# Then upgrade DataHub plugin
pip install --upgrade acryl-datahub-airflow-plugin

Option 2: Stay on Older Plugin Version

If you cannot upgrade Airflow, stay on the previous plugin version:

# Pin to pre-0.3.13.2 version
pip install "acryl-datahub-airflow-plugin<0.3.13.2">

For Docker-based Airflow:

FROM apache/airflow:2.7.3-python3.10

USER airflow

# Install DataHub plugin
RUN pip install acryl-datahub-airflow-plugin==0.3.13.2

Checking Your Current Versions:

# In Airflow Python environment
import airflow
print(f"Airflow version: {airflow.__version__}")

import datahub_airflow_plugin
print(f"DataHub plugin version: {datahub_airflow_plugin.__version__}")

What to Expect if Running Incompatible Versions:

Likely Issues: - Import errors for Airflow modules - DAG parsing failures - Lineage emission failures - Unexpected exceptions during task execution - Missing or incomplete metadata in DataHub

Error Examples: - ImportError: cannot import name 'X' from 'airflow.Y' - AttributeError: module 'airflow' has no attribute 'Z' - Plugin initialization errors in Airflow logs

Testing Compatibility:

After upgrading, verify the plugin works:

# test_datahub_plugin.py
from datahub_airflow_plugin.hooks.datahub import DatahubRestHook

# Try to initialize the hook
hook = DatahubRestHook(datahub_conn_id="datahub_rest_default")
print("✓ DataHub plugin loaded successfully")

# Test a simple lineage emission
from airflow import DAG
from datahub_airflow_plugin.operators.datahub import DatahubEmitterOperator

# If no errors, plugin is compatible
print("✓ Plugin is compatible with your Airflow version")

Migration Checklist:

• [ ] Check current Airflow version
• [ ] Review Airflow 2.7 breaking changes if upgrading
• [ ] Test Airflow upgrade in development environment
• [ ] Backup Airflow metadata database
• [ ] Upgrade Airflow to 2.7.0 or higher
• [ ] Upgrade DataHub plugin to latest version
• [ ] Test DAG parsing and execution
• [ ] Verify lineage appears in DataHub
• [ ] Monitor Airflow logs for errors

• For managed Airflow services (AWS MWAA, Cloud Composer), check provider documentation for Airflow 2.7 support
• If staying on Airflow < 2.7, do not upgrade the DataHub plugin
• Consider this upgrade as part of regular Airflow maintenance
• Airflow 2.7+ includes performance improvements and new features

• DataHub Airflow Integration
• Airflow Upgrade Documentation

• 4975