Skip to main content

Airflow Plugin 1.5.x Silent Failure Due to Missing openlineage-airflow Dependency

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Airflow Plugin Installation & Dependency Management

Issue

Starting with acryl-datahub-airflow-plugin version 1.5.x, the plugin unconditionally imports the openlineage module at load time as part of the v2 lineage listener path. However, the OpenLineage packages (openlineage-airflow for Airflow 2.x, or apache-airflow-providers-openlineage for Airflow 3.x) are not declared as hard install dependencies in the base package — they are only pulled in when the appropriate extras ([airflow2] or [airflow3]) are specified at install time. When a user installs the plugin via a bare pip install acryl-datahub-airflow-plugin — or through a managed environment UI such as Google Cloud Composer's pypi_packages interface that does not support PEP 508 extras syntax — the missing import causes Airflow's plugin manager to mark the plugin as broken and silently disable it entirely. DAGs continue to run normally, but no metadata or lineage is emitted to DataHub, making this failure mode difficult to detect without actively inspecting Airflow worker logs.

Error Messages

  • Broken plugin: [datahub_airflow_plugin.datahub_plugin] No module named 'openlineage'

You Might Be Asking

  • Why are my DAGs running successfully but no lineage or metadata is appearing in DataHub?
  • Why does the DataHub Airflow plugin appear to install without errors but produce no output?
  • How do I install the DataHub Airflow plugin on Google Cloud Composer when the UI does not support package extras syntax?
  • Did something change in the DataHub Airflow plugin 1.5.x that broke my existing integration?

Solution

  1. Verify whether the plugin is loaded by inspecting Airflow scheduler and worker logs. Look for a line containing Broken plugin referencing datahub_airflow_plugin: 
    Broken plugin: [datahub_airflow_plugin.datahub_plugin] No module named 'openlineage'
    If this line is present, the plugin is not active and no metadata is being sent to DataHub.
  2. Use the correct extras-based install command (preferred, where supported). In environments where PEP 508 extras syntax is available (standard pip, requirements.txt, Poetry, etc.), install the plugin with the appropriate extra for your Airflow version:
    • For Airflow 2.x: 
      pip install 'acryl-datahub-airflow-plugin[airflow2]'
      This pulls in openlineage-airflow>=1.2.0 as a transitive dependency.
    • For Airflow 3.x: 
      pip install 'acryl-datahub-airflow-plugin[airflow3]'
      This pulls in apache-airflow-providers-openlineage>=1.0.0.
  3. Workaround for managed environments that do not support extras syntax (e.g., Google Cloud Composer). If your environment's package management UI does not accept PEP 508 extras syntax, install the OpenLineage dependency as a separate, explicit package entry alongside the DataHub plugin. In Composer's pypi_packages configuration, add both entries: 
    acryl-datahub-airflow-plugin
openlineage-airflow>=1.2.0
    This is functionally equivalent to using the [airflow2] extra and resolves the missing import at plugin load time.
  4. Validate the fix by confirming the plugin loads successfully. After restarting the Airflow scheduler and workers (or after a Composer environment update completes), inspect the logs and confirm the broken plugin message is no longer present. Then trigger a DAG and verify that lineage and metadata appear in the DataHub UI.
  5. Plan migration to Airflow 3.x. Apache Airflow 2.x has reached end of life. The DataHub Airflow plugin team has deprecated active support for Airflow 2.x. On Airflow 3.x, the OpenLineage provider package is installed automatically as part of the [airflow3] extras path, and this class of silent failure is not expected to occur. Migrating to Airflow 3.x is the long-term resolution.

Additional Notes

This behavior was introduced in acryl-datahub-airflow-plugin 1.5.x when Airflow 3.x support was added. Prior to this release, openlineage-airflow was declared as a hard base dependency and was always pulled in transitively on a bare pip install. The dependency was intentionally moved to version-specific extras ([airflow2] and [airflow3]) because Airflow 2 and Airflow 3 require mutually incompatible OpenLineage packages that cannot both exist in install_requires. The documented installation path for 1.5.x and later is the extras-based install. The silent failure mode — where the plugin is marked broken with no actionable warning surfaced to operators — is a known defect tracked by the engineering team for a future fix that would guard the load-time imports and emit a clear warning message. Note also that Airflow does not propagate broken plugin state to DAG task outcomes, so DAG runs will appear successful even when the plugin is completely non-functional. Always verify metadata emission in the DataHub UI after any new plugin installation.

Related Documentation

Tags: airflow-plugin, openlineage, silent-failure, broken-plugin, cloud-composer, airflow2, ingestion, dependency, installation, lineage