Airflow Plugin Causing Scheduler Stalls and Timeout Failures When GMS Is Slow or Unavailable – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Airflow Plugin Configuration and Stability

Issue

When the DataHub Airflow plugin is installed and GMS (the DataHub metadata service) is slow to respond or briefly unavailable — due to database connection pool exhaustion, lock contention on high-volume write paths, or infrastructure events such as rolling pod restarts — the Airflow scheduler process can stall, miss liveness heartbeats, and be killed by Kubernetes. This occurs because older versions of the plugin emit metadata synchronously on the scheduler's main execution path: each DAG run start triggers a blocking HTTP POST to /gms/aspects?action=ingestProposal and waits for a response within the configured read timeout. If GMS does not respond within that window, the scheduler thread is held, retries compound the stall, and the scheduler eventually appears unhealthy. The problem affects both the Airflow scheduler process and, under certain configurations, worker processes as well.

Error Messages

ReadTimeoutError("HTTPSConnectionPool(host='<your-instance>.acryl.io', port=443): Read timed out. (read timeout=30.0)")
Max retries exceeded with url: /gms/aspects?action=ingestProposal

You Might Be Asking

Why does setting DATAHUB_EMIT_MODE=ASYNC not prevent the scheduler from stalling?
Why are read-only endpoints like GET /gms/config healthy while POST /gms/aspects?action=ingestProposal times out?
Is it safe to set DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=0 on the scheduler?
What is the difference between the scheduler and worker behavior with respect to emit threads?
Will upgrading the plugin version fix this permanently?

Root Cause Explanation

There are two distinct but related causes that produce identical timeout symptoms:

GMS database write-path saturation: Under high metadata write volume, the metadata_aspect_v2 table can trigger SELECT ... FOR UPDATE row-level locks that hold progressively longer as the table grows. Past a threshold, queued write threads exhaust the GMS database connection pool. All POST /gms/aspects?action=ingestProposal requests stall because connections are accepted but GMS cannot complete the write and return a response. Read-only endpoints (e.g., GET /gms/config) remain healthy because they do not participate in the write lock chain. Spurious 401 Unauthorized errors during such events are also caused by pool exhaustion (GMS cannot query the token store), not by an authentication misconfiguration.
Brief GMS unavailability during infrastructure events: Rolling node restarts, pod rescheduling, or other automated infrastructure operations can briefly take GMS replicas offline. If anti-affinity is not configured across GMS replicas, multiple replicas may restart within a short window, causing a gap during which all write requests time out. The plugin's synchronous emit behavior means the scheduler blocks during this gap.

Why `DATAHUB_EMIT_MODE=ASYNC` Alone Is Insufficient

Setting DATAHUB_EMIT_MODE=ASYNC instructs GMS to queue the write asynchronously on its side once the request is received. It does not make the HTTP call from the plugin non-blocking. The plugin still opens a synchronous HTTP POST to /aspects?action=ingestProposal and waits for GMS to acknowledge the request. If GMS is down or saturated and cannot respond at all, the plugin's POST sits until the read timeout fires, regardless of emit mode. ASYNC helps reduce GMS-side write latency under load, but it cannot protect the scheduler when GMS is unreachable.

Solution

Apply the following mitigations in order of immediacy. Mitigations 1–3 address scheduler protection; mitigation 4 is the recommended long-term fix.

Set DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=0 on the Airflow scheduler only.
This causes the plugin to fire the emit in a background thread and immediately continue without waiting for the result. The scheduler is no longer blocked if GMS is slow or unavailable.

Add the following environment variable to your Airflow scheduler deployment only (not workers — see the warning below):
```
DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=0
```
Restart the scheduler after applying this change.

⚠️ Warning — Airflow version compatibility: This setting behaves differently across Airflow major versions. On Airflow 2, setting this to 0 on workers is dangerous: a worker process exits after each task and kills any background emit thread, which can cause data loss or errors. On Airflow 3, the same risk exists on workers. Apply RUN_IN_THREAD_TIMEOUT=0 only to the scheduler process. The scheduler does not terminate between tasks, so background emit threads complete safely.
Also set DATAHUB_EMIT_MODE=ASYNC.
While this does not eliminate the scheduler block, it reduces GMS-side write latency under load, which shortens the window during which timeouts can occur. Apply it as an environment variable on both the scheduler and workers.
```
DATAHUB_EMIT_MODE=ASYNC
```
Note: The emit_mode: ASYNC config file option was not introduced until plugin version 1.6.0.16. For earlier versions, the environment variable is the only supported method.
Tune Airflow connection timeout and retry settings.
The default plugin settings (read timeout 30 s, connect timeout 30 s, up to 4 retries) cause the scheduler to stall for up to two minutes per event when GMS is slow. Reducing these values limits the blast radius of any single GMS slowdown.

In the Airflow UI, open your DataHub connection, navigate to the Extra field, and set:
```
{
  "retry_max_times": 1,
  "read_timeout_sec": 10,
  "connect_timeout_sec": 5
}
```
With retry_max_times: 1 and read_timeout_sec: 10, the maximum stall per emit event is approximately 20 seconds rather than 240 seconds at the defaults.
Upgrade to acryl-datahub-airflow-plugin version 1.7.0.1 or later (recommended long-term fix).
In plugin version 1.7.0.1 and later, ASYNC emit mode is the default behavior. This removes the need for the DATAHUB_EMIT_MODE=ASYNC environment variable workaround. Combined with the RUN_IN_THREAD_TIMEOUT=0 scheduler setting, upgrading provides the most resilient configuration against GMS slowness or brief unavailability.
```
pip install "acryl-datahub-airflow-plugin>=1.7.0.1"
```

Verifying the Fix

Monitor Airflow scheduler heartbeat metrics. After applying RUN_IN_THREAD_TIMEOUT=0, heartbeat dips caused by GMS timeouts should stop.
Confirm that DAG metadata continues to appear in DataHub after applying the non-blocking configuration. Emit threads complete in the background; successful task runs should still result in lineage and run data being recorded.
If GMS experiences another slow period, the scheduler process should remain healthy and continue scheduling DAGs normally.

Summary of Environment Variable Configuration

# Apply to the Airflow SCHEDULER only
DATAHUB_AIRFLOW_PLUGIN_RUN_IN_THREAD_TIMEOUT=0

# Apply to both the SCHEDULER and WORKERS
DATAHUB_EMIT_MODE=ASYNC

Additional Notes

GMS-side root causes require server-side remediation. The client-side mitigations described above protect the Airflow scheduler from being impacted by GMS slowness, but they do not prevent GMS database connection pool exhaustion or resolve lock contention in the write path. If you observe GMS saturating its connection pool or metadata_aspect_v2 write locks cascading, contact DataHub Support to investigate the GMS configuration, connection pool sizing, and write path locking behavior for your instance.

Infrastructure anti-affinity for GMS replicas. If your DataHub deployment runs multiple GMS replicas (whether self-hosted or managed), ensure that pod anti-affinity rules are configured so that replicas are not co-located on the same node. Without anti-affinity, a single automated node rotation can reschedule all GMS pods simultaneously, causing a complete write outage that is indistinguishable from pool exhaustion at the client. This is a Kubernetes deployment-level configuration.

RUN_IN_THREAD_TIMEOUT=0 is not recommended on Airflow worker processes. Workers exit after each task finishes. A background emit thread started by a worker will be killed when the worker exits if the thread has not yet completed. This can result in metadata not being emitted for that task. Use RUN_IN_THREAD_TIMEOUT=0 only on the scheduler process.

Spurious 401 errors during GMS saturation events are not authentication failures. They occur because GMS cannot service token lookups when the database connection pool is exhausted. No credential or token rotation is required in response to these errors; they resolve when GMS recovers.

Plugin version compatibility summary:

1.4.x: Use DATAHUB_EMIT_MODE=ASYNC environment variable; config file option not available. Default emit is synchronous and blocking.
1.6.0.16+: emit_mode: ASYNC config file option available in addition to the environment variable.
1.7.0.1+: ASYNC emit is the default; no environment variable needed for emit mode. Recommended minimum version for Airflow deployments running at scale.