Resolving Airflow Plugin Compatibility Issues with Version Upgrades – DataHub Customer Support Portal

Area: Deployment Issues

Sub-Area: Airflow Plugin Integration

Issue

DataHub Airflow plugin versions 1.1.0+ cause Airflow environments to become unstable, resulting in scheduler crashes, worker startup failures, and heartbeat drops to zero. This commonly occurs after upgrading the plugin or when dependency managers automatically pull newer versions during environment updates.

Error Messages

Thread for on_dag_run_running is still running after 10.0 seconds. Continuing without waiting for it to finish. [datahub_airflow_plugin.airflow3.datahub_listener]
AttributeError: 'NoneType' object has no attribute 'split'
Celery worker initialization errors

You Might Be Asking

Why did my Airflow scheduler stop responding after upgrading DataHub plugin?
How do I fix Airflow workers that won't start after adding the DataHub plugin?
What DataHub plugin version should I use with my Airflow version?

Solution

Immediate Workarounds

Pin to a compatible version: Based on testing, pin to version 1.0.0.3 which is known to work:
```
acryl-datahub-airflow-plugin==1.0.0.3
```
For latest plugin versions, use minimal installation: Install without extras to avoid dependency conflicts:
```
acryl-datahub-airflow-plugin
```
Do NOT use [airflow2], [plugin-v2], or other extras.
Increase thread timeout: For scheduler crashes, increase the timeout:
```
export DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=60
```
Disable threading: Make operations synchronous (may slow DAG starts):
```
export DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD=false
```
Temporarily disable plugin: For critical environments:
```
export DATAHUB_AIRFLOW_PLUGIN_ENABLED=false
```

Configuration Adjustments

If using many concurrent DAGs, reduce Airflow scheduler load:

AIRFLOW__SCHEDULER__MAX_DAGRUNS_PER_LOOP_TO_SCHEDULE=
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY=

For Google Cloud Composer Users

Avoid using plugin extras that conflict with Composer's built-in OpenLineage
Set environment variables in Composer's environment configuration
Test plugin changes in a non-production environment first

Additional Notes

Plugin versions 1.1.0+ introduced threading changes and OpenLineage integration that can interfere with Celery worker initialization, especially in managed Airflow environments like Google Cloud Composer. The plugin's initialization timing can conflict with Airflow's startup sequence. For environments with high DAG concurrency, the plugin may create too many background threads, starving the scheduler of resources. Always test plugin upgrades in development environments before deploying to production.