Reducing High Ingestion Duration Caused by Write Rate Limiting Using Async Emit Mode – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Custom Ingestion Performance / Write Throughput

Issue

Custom ingestion scripts that emit a high volume of metadata writes in synchronous mode can trigger backend write rate limiting on DataHub Cloud instances. When rate limiting is active, the DataHub SDK automatically waits and retries each rejected write, causing ingestion run durations to increase significantly — sometimes across all custom ingestion sources simultaneously — without any loss of data correctness. This pattern is particularly common when scripts perform large numbers of individual synchronous writes rather than batched or asynchronous emissions.

You Might Be Asking

Why did my custom ingestion scripts suddenly start taking much longer to complete, with no changes on my side?
Why are ingestion runs slow even though all data appears to be ingesting correctly?
What does write rate limiting in DataHub Cloud mean, and how does it affect my ingestion scripts?
How do I switch my DataHub ingestion scripts to async emit mode?
Will switching to DATAHUB_EMIT_MODE=ASYNC cause data loss or consistency issues?

Solution

Identify whether write rate limiting is the cause.
Check the following indicators in your ingestion run logs and environment:
- Ingestion duration has increased significantly across custom scripts, but data correctness is not affected.
- No changes were made to the ingestion scripts, source systems, or infrastructure around the time the slowdown began.
- The scripts emit a large number of synchronous metadata writes per run.
- Individual write operations are slower than expected, consistent with retry/backoff behavior in the SDK.
If your DataHub Cloud instance has rate limiting applied to metadata writes, the SDK will silently retry rejected writes, prolonging run time. Contact DataHub Support if you suspect rate limiting is active on your instance.
Switch your ingestion scripts to async emit mode.
Set the following environment variable in the environment where your custom ingestion scripts run:
```
DATAHUB_EMIT_MODE=ASYNC
```
In async mode, the DataHub backend accepts each write immediately and persists it shortly after. This significantly reduces instantaneous load on the backend database compared to synchronous mode, which requires each write to be fully committed before the next one is accepted.

You can set this variable in your shell, in a .env file, or in your container/pod environment configuration, depending on how your scripts are deployed. For example, in a shell:
```
export DATAHUB_EMIT_MODE=ASYNC
python your_ingestion_script.py
```
Or inline for a single run:
```
DATAHUB_EMIT_MODE=ASYNC python your_ingestion_script.py
```
Batch your metadata emissions using emit_mcps.
In addition to enabling async mode, restructure your scripts to send metadata changes in batches rather than one write at a time. Using the emit_mcps method, a collection of Metadata Change Proposals (MCPs) is sent as a single request, which counts as one operation against any rate limit and reduces network overhead.

Example using the DataHub Python SDK:
```
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.emitter.mcp import MetadataChangeProposalWrapper

emitter = DatahubRestEmitter(gms_server="https://.datahubproject.io/api/gms")

mcps = []
for entity in entities_to_emit:
    mcp = MetadataChangeProposalWrapper(
        entityUrn=entity["urn"],
        aspect=entity["aspect"],
    )
    mcps.append(mcp)

# Emit all MCPs in a single batched request
emitter.emit_mcps(mcps)
```
Adjust batch size based on your use case. Smaller batches (e.g., 100–500 MCPs) are a reasonable starting point for high-volume scripts.
Account for eventual consistency when reading back written data.
In async mode, if your script writes an entity and then immediately reads it back, it may briefly observe the previous value. Design your scripts to tolerate this short propagation delay. If your workflow requires reading the written value immediately after writing it, consider adding a short wait or restructuring the workflow to separate write and read phases.
Verify that ingestion duration returns to normal.
After deploying the async emit mode change, re-run your ingestion scripts and compare run durations against historical baselines. If rate limiting was applied to your instance by DataHub Cloud, notify DataHub Support that you have switched to async mode so that any active write rate limit can be reviewed and lifted if appropriate.

Additional Notes

Write rate limiting may be applied temporarily by DataHub Cloud operations teams when a specific instance's synchronous write volume causes elevated database load. The limit affects write throughput (requests per second) but does not cause data loss — the DataHub SDK retries rejected writes automatically. The rate limit is typically lifted once the instance's write pattern shifts to async mode and load normalizes. If you believe rate limiting is currently active on your instance, contact DataHub Support for confirmation and to coordinate removal of the limit.

Async emit mode is the recommended configuration for any high-volume custom ingestion workload on DataHub Cloud, regardless of whether rate limiting is currently active. It reduces backend database pressure, improves script throughput, and makes your ingestion more resilient to transient backend load spikes.

If your scripts install Python packages at runtime without pinned versions, an unexpected package update could also affect ingestion duration independently of rate limiting. Pin all dependencies in your ingestion environment to avoid unintended behavior changes.

This behavior applies to DataHub Cloud (managed) deployments. Self-hosted DataHub users who experience similar symptoms should review GMS and Kafka consumer lag metrics to identify bottlenecks.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation