Rate Limiting in DataHub Cloud: Best Practices for Avoiding HTTP 429 Errors – DataHub Customer Support Portal

Area: API Issues

Sub-Area: Rate Limiting and Throttling

Issue

When emitting metadata to DataHub Cloud at high volumes — whether via the DataHub CLI, Python SDK, or automated ingestion pipelines — clients may receive HTTP 429 (Too Many Requests) errors from the GMS (Metadata Service) endpoint. This occurs when ingestion volume crosses the platform's built-in rate limiting thresholds, causing the API gateway to throttle requests and instruct clients to back off for a calculated wait period. The DataHub CLI will automatically retry up to a configured number of times, but if the throttle condition persists across all retries, it raises an OperationalError and the ingestion job fails. This is intentional behavior designed to protect system stability and ensure fair resource usage across all tenants in DataHub Cloud.

Error Messages

datahub.configuration.common.OperationalError: ('Unable to emit metadata to DataHub GMS: Throttled due to [ThrottleEvent(backoffWaitMs={RATE_LIMIT=<ms>}, jitterRatio=2.0)]', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'message': 'Throttled due to [ThrottleEvent(backoffWaitMs={RATE_LIMIT=<ms>}, jitterRatio=2.0)]', 'status': 429})

You Might Be Asking

Why is my DataHub CLI returning HTTP 429 errors when emitting metadata?
What are the rate limit thresholds in DataHub Cloud?
How do I prevent my ingestion pipelines from being throttled by DataHub Cloud?
Does each retry count as a separate API call against the rate limit?
Can DataHub Cloud rate limit thresholds be increased for my deployment?

Solution

The following steps explain the rate limiting behavior and how to configure your client-side tooling and ingestion workflows to avoid sustained throttling.

Step 1: Understand the Rate Limiting Thresholds

DataHub Cloud enforces two conditions that together trigger rate limiting on the GMS endpoint:

Activation threshold: Your instance has emitted more than 100,000 metadata events in the past 24 hours.
Rate limit threshold: The current 5-minute rolling window contains more than 30,000 metadata events.

Both conditions must be true simultaneously for throttling to activate. When throttling triggers, the server calculates a backoff wait time (reflected in the backoffWaitMs value in the error) plus a jitter multiplier (default: 2.0x) to prevent synchronized retry storms. This means the effective client wait can be up to twice the reported backoff value.

Important: Every API call — including each individual retry attempt — counts separately against the rate limit. Aggressive retry loops without proper backoff can worsen the throttle condition rather than resolve it.

Step 2: Wait for the Throttle to Self-Clear

If you are actively hitting the rate limit, the fastest immediate resolution is to pause all ingestion and wait for the 5-minute rolling window to drain below 30,000 events. Once the window clears, re-run the failed ingestion job. This is the recommended first step before making any configuration changes.

Step 3: Implement Exponential Backoff and Retries

Configure your ingestion clients to use exponential backoff on 429 responses. The DataHub REST emitter exposes environment variables for tuning retry behavior:

# Maximum number of retry attempts on failure (default: 4)
export DATAHUB_REST_EMITTER_DEFAULT_RETRY_MAX_TIMES=4

# Backoff multiplier applied between retries on 429 responses (default: 2)
export DATAHUB_REST_EMITTER_429_RETRY_MULTIPLIER=2

A recommended starting configuration is 3–4 retries with an exponential backoff multiplier of 2. This allows the rolling window time to drain between attempts without flooding the API with rapid successive calls.

If you are using the Python SDK directly in a custom script, implement backoff logic explicitly:

import time
import random
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter(gms_server="https://<your-instance>.acryl.io/gms")

def emit_with_backoff(mcp, max_retries=4, base_wait=10):
    for attempt in range(max_retries):
        try:
            emitter.emit(mcp)
            return
        except Exception as e:
            if "429" in str(e) or "RATE_LIMIT" in str(e):
                wait = base_wait * (2 ** attempt) + random.uniform(0, base_wait)
                print(f"Rate limited. Waiting {wait:.1f}s before retry {attempt + 1}/{max_retries}")
                time.sleep(wait)
            else:
                raise
    raise RuntimeError("Max retries exceeded due to rate limiting.")

Step 4: Reduce Ingestion Parallelism and Batch Size

If multiple processes or pipelines are running concurrently and collectively contributing to the event count, reduce parallelism to lower the events-per-minute rate:

Run ingestion jobs sequentially rather than in parallel where possible.
Introduce deliberate delays between batches of metadata emission calls.
Break large bulk ingestion jobs into smaller chunks with pauses between each chunk.

Example: adding a small sleep between batches in a custom emission loop:

import time

BATCH_SIZE = 500
INTER_BATCH_DELAY_SECONDS = 5  # Adjust based on your event volume

mcps = [...]  # Your full list of MetadataChangeProposalWrappers

for i in range(0, len(mcps), BATCH_SIZE):
    batch = mcps[i:i + BATCH_SIZE]
    for mcp in batch:
        emitter.emit(mcp)
    print(f"Emitted batch {i // BATCH_SIZE + 1}. Sleeping {INTER_BATCH_DELAY_SECONDS}s...")
    time.sleep(INTER_BATCH_DELAY_SECONDS)

Step 5: Stagger Ingestion Schedules

If you have multiple ingestion sources or pipelines scheduled to run on a recurring basis, avoid scheduling them to start at the same time. Stagger start times so that peak event emission from different sources does not overlap within the same 5-minute window.

For example, in a cron-based scheduler:

# Instead of running all jobs at midnight:
# 0 0 * * *  pipeline_a
# 0 0 * * *  pipeline_b
# 0 0 * * *  pipeline_c

# Stagger them across a 30-minute window:
0 0 * * *   pipeline_a
15 0 * * *  pipeline_b
30 0 * * *  pipeline_c

Step 6: Audit All Sources Contributing to Event Volume

The 24-hour and 5-minute event counts accumulate across all sources emitting to your DataHub Cloud instance — this includes native DataHub ingestion connectors, CLI-based ingestion, custom SDK integrations, and any CI/CD pipelines that push metadata. If you are unsure which sources are contributing the most events, audit all processes that call the GMS API and estimate their event volumes. Prioritize reducing or rate-limiting the highest-volume sources first.

Additional Notes

Rate limiting in DataHub Cloud is enforced at the API gateway layer and is intended to protect platform stability for all tenants. The thresholds are not tied to performance tiers or contract levels — they apply uniformly to protect infrastructure. The default thresholds are:

Activation quota: 100,000 events per 24-hour window
Rate limit per interval: 30,000 events per 5-minute window
Interval duration: 300 seconds (5 minutes)

These thresholds are configured via server-side environment variables (MCP_RATE_LIMIT_ACTIVATION_QUOTA, MCP_RATE_LIMIT_PER_INTERVAL, MCP_RATE_LIMIT_INTERVAL_SECONDS) and are not adjustable by end users. If your workload legitimately and consistently exceeds these thresholds after applying client-side optimizations, contact DataHub Support to discuss your ingestion volumes and explore available options. In most cases, reducing parallelism, implementing proper backoff, and staggering schedules is sufficient to operate within the limits without impacting throughput significantly.

The backoffWaitMs value in the error message indicates the server-calculated wait time in milliseconds. The effective maximum wait (including jitter) is backoffWaitMs × jitterRatio (default jitter: 2.0x). Plan retry intervals accordingly.