Airflow DataHub Lineage Plugin: Rate Limiting Behavior and Log Configuration – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Airflow Plugin Configuration

Issue

When integrating the acryl-datahub-airflow-plugin to emit lineage events from Airflow DAGs using inlets and outlets, users often have questions about two specific behaviors: whether the plugin enforces any rate limiting when calling the DataHub API, and whether plugin logs or API errors are routed through DataHub's backend (RE) pods or are only visible within the Airflow environment itself.

You Might Be Asking

Does the Airflow DataHub lineage plugin have built-in rate limiting?
What happens if the DataHub API returns an error during lineage emission — will I see those errors server-side?
Where do I find logs from the Airflow DataHub plugin when lineage emission fails?
Does the plugin route requests through the DataHub RE (Reporting Engine) pods?

Solution

Rate Limiting: The acryl-datahub-airflow-plugin itself does not impose any plugin-level rate limiting. Rate limiting, if encountered, would originate from the DataHub GMS (Generalized Metadata Service) layer, not the plugin. The plugin does, however, include retry and backoff mechanisms inherited from the shared base emitter class (rest_emitter.py). This means transient API errors will automatically be retried with exponential backoff before a failure is surfaced.
The relevant retry/backoff logic in the base REST emitter looks like this:
```
# From datahub/emitter/rest_emitter.py
# Retry with backoff is handled automatically by the base class.
# No additional rate-limit configuration is required in the plugin itself.
```
You do not need to configure any rate-limiting parameters in the plugin. If you are hitting GMS-level rate limits, contact your DataHub administrator to review GMS throughput settings.
Log Routing and Observability: Plugin logs are not routed through the DataHub RE (Reporting Engine) pods. The plugin runs entirely within your Airflow environment, so any API errors or lineage emission failures will only appear in your Airflow scheduler process logs — not in DataHub server-side logs.
To capture and inspect these logs, configure the log level for the DataHub plugin within your airflow.cfg file. For example:
```
[logging]
# Set the log level for the DataHub plugin logger
# Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
logging_level = INFO
```
You can also set plugin-specific log levels using the optional configurations documented in the DataHub Airflow lineage docs (see Related Documentation below). Logs will surface in your Airflow scheduler process output.
Summary of Expected Behavior:
- No plugin-level rate limiting is enforced — you can emit lineage events freely from your DAGs.
- Retry and backoff on API failures are handled automatically by the base emitter class.
- All plugin logs appear in the Airflow scheduler process, not in DataHub RE pods.
- If lineage emission fails due to an API error, there will be no corresponding server-side log in DataHub — you must monitor Airflow logs for failures.

Additional Notes

Because plugin logs are only available within the Airflow scheduler process, it is strongly recommended to configure an appropriate log level (e.g., INFO or DEBUG) in your Airflow environment during initial setup and testing of the lineage plugin. This ensures that any emission errors are visible and actionable. Once the integration is stable, you may reduce the log verbosity. The retry/backoff behavior of the base REST emitter is shared across all DataHub ingestion sources, so it applies consistently regardless of how the plugin is configured.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation