Skip to main content

Airflow Plugin: Missing Run History Due to DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT Setting

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Airflow Plugin Configuration

Issue

When using the DataHub Airflow plugin, pipeline run history may not appear in the Runs tab for Airflow data jobs in the DataHub UI. Tasks may appear stuck in a Running state indefinitely, and no completed or failed run records are written to DataHub. This occurs even when the plugin is installed, enabled, and Airflow assets are otherwise visible in DataHub. The root cause is the DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT setting being configured to 0, which causes Airflow to immediately abandon the background thread responsible for emitting run metadata to DataHub before it has a chance to complete its API calls.

Error Messages

  • No run history visible in DataHub UI for Airflow pipeline tasks
  • Task run status remains "Running" indefinitely in DataHub

You Might Be Asking

  • Why does the Runs tab show nothing for my Airflow pipelines even though lineage and assets appear correctly?
  • Why are my Airflow task runs stuck in "Running" status in DataHub?
  • Does setting DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT to 0 prevent run metadata from being emitted?
  • Will this setting cause Airflow scheduler stalling?

Solution

  1. Locate your DataHub Airflow plugin configuration. Depending on your Airflow deployment, this may be in an airflow.cfg file or set via environment variables in your Airflow management platform (e.g., Astronomer, MWAA, Composer).

  2. Check the current value of DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT. A value of 0 means Airflow will not wait at all for the background emission thread to complete, causing all run metadata to be silently dropped:

    [datahub]
enabled = True
platform_instance = <your-platform-instance>
cluster = <your-cluster>
render_templates = False
DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT = 0   # ← This is the problematic setting

  3. Remove the DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT setting entirely, or set it to the recommended default value of 10 (seconds). This allows the background thread sufficient time to complete its API calls to DataHub before Airflow moves on:

    [datahub]
enabled = True
platform_instance = <your-platform-instance>
cluster = <your-cluster>
render_templates = False
DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT = 10

  4. If you manage configuration via environment variables rather than airflow.cfg, update or remove the corresponding environment variable in your deployment platform. For example:

    # Remove or update this environment variable in your deployment
DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=10

  5. Restart your Airflow deployment (scheduler and workers) to apply the configuration change.

  6. Trigger a test DAG run and then verify that run history begins appearing in the DataHub UI under the Runs tab for the corresponding data job. Allow a few minutes for the metadata to propagate.

Additional Notes

The DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT setting controls how long (in seconds) Airflow will wait for the DataHub plugin's background emission thread to finish before the Airflow process continues. Setting this value to 0 is sometimes applied as a defensive measure to prevent Airflow scheduler stalling, but it has the unintended consequence of completely preventing run metadata (task start, success, failure events) from being written to DataHub. The default value is 10 seconds, which is sufficient for the plugin to complete its API calls under normal network conditions. If you are concerned about scheduler stalling, refer to the DataHub documentation on scheduler stalling for guidance on tuning this value appropriately rather than disabling it entirely. This behavior applies to all versions of the DataHub Airflow plugin (both the [airflow2] and [airflow3] extras). Note that Airflow asset lineage and job metadata may still appear correctly in DataHub even when run history is missing, since those are emitted via a different mechanism.

Related Documentation

Tags: airflow, airflow-plugin, run-history, missing-runs, DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT, timeout, configuration, lineage, ingestion, astronomer