GMS Timeouts and OOM Crashes Caused by Oversized Data Product Asset Lists – DataHub Customer Support Portal

Area: Observability Issues

Sub-Area: GMS Performance / Data Products API at Scale

Issue

When a DataHub Data Product accumulates a very large number of linked assets (thousands of entries), every write to that Data Product causes GMS (General Metadata Service) to serialize and transmit the full asset list. If multiple writes target the same Data Product concurrently — for example, because a pipeline adds one new asset per run — those writes contend on the same database row, each holding a GMS Jetty thread for 10–30 seconds. Under sustained concurrent load, all available Jetty threads become saturated, GMS heap memory is exhausted, and GMS pods crash with OOM (Out of Memory) restarts. Callers experience HTTP timeouts on the /gms/aspects?action=ingestProposalBatch endpoint, and upstream jobs that depend on those writes are dropped. The pattern tends to recur on a predictable schedule whenever an automated pipeline runs at a fixed time each day.

Error Messages

POST /gms/aspects?action=ingestProposalBatch — 504 Gateway Timeout
java.lang.OutOfMemoryError: Java heap space (visible in GMS pod logs before restart)
com.linkedin.restli.server.RestLiServiceException: Request timed out after 55000ms

You Might Be Asking

Why does writing to a Data Product with many assets cause GMS to crash?
Why do I see GMS pod OOM restarts at the same time every day?
Why are my ingestProposalBatch calls timing out even though each individual write seems small?
How can I add assets to a Data Product without causing GMS performance problems?
Does setting async=true on ingestProposalBatch prevent these timeouts?

Solution

Step 1 — Immediate Recovery

Identify and stop the ingestion source. Find which pipeline or automated job is writing to the large Data Product at a recurring time (check Airflow DAGs, cron schedules, or SDK-based scripts). Pause or stop that job to allow GMS Jetty threads to drain.
Restart GMS pods if the service does not self-recover. After the ingestion source is stopped, if GMS is still unresponsive, restart the GMS pods. This releases any pinned or leaked threads and clears heap pressure.
Confirm recovery. Verify that /gms/aspects?action=ingestProposalBatch returns within normal latency before resuming ingestion.

Step 2 — Root Cause: Identify Oversized Data Products

A Data Product becomes a performance liability when it accumulates thousands of linked assets, especially when those assets include one entry per pipeline run (for example, one metadata mapping job created per execution). Use the DataHub GraphQL API or REST API to inspect asset counts for your largest Data Products:

# GraphQL query — count assets on a Data Product
query {
  dataProduct(urn: "urn:li:dataProduct:<your-data-product-id>") {
    entities(input: { start: 0, count: 0 }) {
      total
    }
  }
}

Data Products with more than a few thousand assets are candidates for the OOM pattern described here. Pay particular attention to Data Products that grow unboundedly because a new child entity is created for every pipeline run.

Step 3 — Stop Adding Per-Run Job Entries to Data Products

The most effective long-term fix is to avoid linking ephemeral, per-run entities (such as DataProcessInstance run records) directly to a Data Product. Instead:

Link one stable, long-lived job or pipeline entity to the Data Product, rather than a new entity for every execution.
Use DataProcessInstance run events for lineage and observability purposes without attaching each instance to the Data Product's asset list.

Step 4 — Remove Existing Stale Per-Run Entries

After changing the ingestion pattern, remove previously accumulated per-run entries from affected Data Products. Use the DataHub Python SDK to remove assets in batches:

from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig
from datahub.metadata.schema_classes import DataProductPropertiesClass

graph = DataHubGraph(DatahubClientConfig(server="https://<your-instance>.datahubproject.io"))

data_product_urn = "urn:li:dataProduct:<your-data-product-id>"

# Fetch current properties
properties = graph.get_aspect(
    entity_urn=data_product_urn,
    aspect_type=DataProductPropertiesClass,
)

# Filter out stale per-run URNs — keep only stable asset URNs
stable_asset_urns = [
    asset for asset in (properties.assets or [])
    if "<stable-asset-identifier>" in asset.destinationUrn
]

properties.assets = stable_asset_urns

graph.emit_mce_async(...)  # emit updated aspect

Step 5 — Use Patch Operations for Incremental Asset Additions

When you do need to add individual assets to a Data Product over time, use the DataProductPatchBuilder patch API instead of overwriting the full asset list. Patch operations send only the diff, not the entire serialized list, which eliminates the multi-megabyte payload problem:

from datahub.specific.data_product import DataProductPatchBuilder

patch_builder = DataProductPatchBuilder(
    urn="urn:li:dataProduct:<your-data-product-id>"
)

# Add a single asset without rewriting the full list
patch_builder.add_asset("urn:li:dataset:(urn:li:dataPlatform:<platform>,<dataset-name>,PROD)")

for mcp in patch_builder.build():
    graph.emit(mcp)

Step 6 — Eliminate Concurrent Writes to the Same Data Product

Concurrent writes to the same Data Product serialize at the database row level, causing each write to block all others and multiply thread hold time. Ensure your pipeline serializes writes to a given Data Product — do not fan out parallel workers that all write to the same Data Product URN simultaneously.

Step 7 — Tune GMS Ingestion Client Settings (Short-Term Relief)

While you restructure your Data Product asset model, reduce Jetty thread pressure with the following environment variable changes on the GMS deployment:

# Reduce per-call batch ceiling to 25 (default is 50)
ENTITY_CLIENT_RESTLI_INGEST_BATCH_SIZE=25

# Serialize ingestion threads to reduce peak Jetty consumption during bursts
ENTITY_CLIENT_RESTLI_INGEST_BATCH_CONCURRENCY=1

# Optional: extend timeout to delay caller errors (does NOT reduce thread saturation)
DATAHUB_GMS_ASYNC_REQUEST_TIMEOUT_MS=90000

Step 8 — Use Async Ingestion Where Possible

For metadata proposals that do not involve timeseries aspects, setting async=true causes GMS to enqueue the proposal to the MCP Kafka topic and return immediately, releasing the Jetty thread without waiting for the database write to complete.

# Python SDK — enable async ingestion on the graph client
from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig

graph = DataHubGraph(DatahubClientConfig(
    server="https://<your-instance>.datahubproject.io",
    extra_headers={},
))

# Pass async_flag=True when emitting proposals
graph.emit(mcp, async_flag=True)

Limitation: Timeseries aspects such as dataProcessInstanceRunEvent still undergo partial synchronous processing on Jetty threads even when async=true is set. This is a known platform limitation. If your pipeline emits large volumes of timeseries aspects, async mode alone may not fully prevent thread saturation.

Additional Notes

The default GMS async request timeout is 55 seconds (DATAHUB_GMS_ASYNC_REQUEST_TIMEOUT_MS=55000). Any single ingestProposalBatch call that holds a Jetty thread for longer than this limit will return a timeout error to the caller. Large Data Product writes (multi-megabyte asset lists) can easily exceed this threshold under database contention.
The OOM crash pattern typically recurs daily at the same time because it is triggered by a scheduled pipeline. Fixing the schedule or the data model is necessary to prevent recurrence; restarting pods alone is not a durable solution.
Data Products with more than approximately 5,000 linked assets should be treated as an architectural signal that the asset-to-data-product mapping strategy needs revision.
A durable GMS-side fix (sync/async-aware admission control for ingestProposalBatch) is tracked internally and has not yet shipped as of the time this article was written. The mitigations described above are the recommended approach until that fix is available.
Jetty thread pool metrics are now exported to observability tooling in recent DataHub Cloud versions. Monitor the jetty.threads.busy and jetty.threads.idle gauges to detect thread saturation before it results in OOM crashes.