Resolving Snowflake Ingestion Failures with HTTP 400 Errors from Large Lineage Payloads – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Snowflake Connector — Lineage Payload Size and Query Entity Generation

Issue

Snowflake ingestion recipes may fail consistently with HTTP 400 errors when DataHub attempts to emit metadata to GMS (Graph Metadata Service). The failures are typically associated with upstreamLineage aspects or query entities for specific datasets or views. This occurs when the generated lineage payload for a given dataset is too large for the GMS server to parse — commonly triggered when a single dataset sits at the root of a deep or wide lineage graph (e.g., 10+ levels of downstream dependencies), causing the serialized aspect to exceed acceptable size limits. A related bug in certain connector versions also causes malformed query entity URNs (URL-encoded dataset URNs embedded in query IDs), which GMS rejects with the same 400 error.

Error Messages

Unable to emit metadata to DataHub GMS: Cannot parse request entity
com.linkedin.restli.server.RestLiServiceException: Cannot parse request entity (status 400)
workunit_id: urn:li:dataset:(...)-upstreamLineage
workunit_id: urn:li:query:view_urn%3Ali%3Adataset%3A%28...%29-dataPlatformInstance

You Might Be Asking

Why does my Snowflake ingestion fail with HTTP 400 for only a few specific assets?
Why does the same recipe succeed in one environment (e.g., a lab or staging instance) but fail in production?
What does "Cannot parse request entity" mean in the context of DataHub GMS?
Why are my query entity URNs URL-encoded and how does that cause ingestion failures?
How can I work around lineage payload failures without disabling lineage entirely?

Solution

There are two distinct but related failure modes. Follow the diagnosis steps below to identify which applies to your environment, then apply the appropriate remediation.

Step 1 — Identify the Failing Assets and Failure Type

Open the failed ingestion run in the DataHub UI and locate the error details. Note the urn and workunit_id fields in each failure entry.
If the workunit_id ends in -upstreamLineage, the failure is caused by an oversized lineage aspect. Proceed to Step 2.
If the workunit_id ends in -dataPlatformInstance and the query URN contains URL-encoded characters (e.g., %3A, %28, %29), the failure is caused by a malformed query entity URN. This is a known connector bug fixed in CLI version 1.5.0 and later. Proceed to Step 4 to upgrade.

Step 2 — Assess the Lineage Depth of the Failing Dataset

For each failing dataset URN, check how many tables or views depend on it downstream (directly or transitively). A dataset with 10 or more levels of downstream lineage is a strong indicator that the serialized upstreamLineage aspect will exceed GMS payload limits.

Optionally, inspect the dataset directly in your Snowflake instance to understand its role:

-- Check view definition size
SELECT GET_DDL('VIEW', '<your_schema>.<your_view_name>');

-- Check tables that reference this dataset
SHOW VIEWS LIKE '<your_view_name>' IN SCHEMA <your_database>.<your_schema>;

Step 3 — Apply Workarounds for Oversized Lineage Payloads

Use one or more of the following workarounds while a permanent fix is deployed.

Option A: Disable Incremental Lineage

Incremental lineage mode accumulates upstream relationships across runs, which can cause the stored aspect to grow unboundedly. Switching to non-incremental (full-snapshot) lineage replaces the aspect on each run instead of appending to it, reducing payload size.

source:
  type: snowflake
  config:
    # ... other config ...
    incremental_lineage: false

Option B: Temporarily Exclude Problematic Assets

If you need the ingestion run to succeed immediately, add the failing datasets to a deny list. You can re-enable them after upgrading to a fixed CLI version or after the lineage graph has been restructured.

source:
  type: snowflake
  config:
    # ... other config ...
    table_pattern:
      deny:
        - "<your_database>\\.<your_schema>\\.<your_table_or_view_name>"

Option C: Increase Remote Executor Pod Resources

If you are running ingestion via a remote executor deployed on Kubernetes, the pod may be running out of memory or ephemeral storage when serializing large lineage payloads. Increase the resource limits for the executor pods:

# Example Kubernetes resource patch for the remote executor worker deployment
# Adjust values to match your environment's capacity
resources:
  requests:
    memory: "2Gi"
    ephemeral-storage: "4Gi"
  limits:
    memory: "4Gi"
    ephemeral-storage: "8Gi"

Apply this to your executor deployment and restart the pods before re-triggering the ingestion run.

Step 4 — Upgrade to a Fixed CLI Version

The malformed query entity URN bug (URL-encoded dataset URNs embedded in urn:li:query: identifiers) was resolved in CLI version 1.5.0. Ensure your ingestion environment is using this version or later.

Verify your current CLI version:
```
datahub version
```
Upgrade to the latest stable release:
```
pip install --upgrade acryl-datahub
```
If you are using a managed or cloud-hosted executor, re-trigger the ingestion run after the platform has been updated to pull the correct CLI image. In some cases, simply re-triggering the run will cause the executor to pull the latest pinned version.

Step 5 — Re-run and Validate

Re-trigger the failing ingestion source after applying the appropriate workaround or upgrade.
Confirm that the previously failing assets now appear in the DataHub UI with correct lineage.
Monitor for any remaining workunit_id failures in the ingestion run summary.

Additional Notes

This issue can appear intermittently if lineage graphs grow over time — an ingestion recipe that previously succeeded may begin failing after a dataset accumulates many downstream dependents across subsequent runs.
The same recipe may behave differently across environments (e.g., lab vs. production) if the two environments have different data or if the lab environment ingests a subset of schemas, resulting in a smaller lineage graph.
The malformed query URN bug (URL-encoded characters in urn:li:query: identifiers) affected all connector types, not only Snowflake. If you observe similar HTTP 400 failures on other source types with urn:li:query: workunits, the same CLI upgrade resolves the issue.
If you are an OSS DataHub user upgrading across major versions, review the updating_datahub documentation for any breaking changes related to stateful ingestion and lineage handling.
Stateful ingestion, when enabled, helps manage deletions and stale lineage entries automatically and is recommended as a best practice alongside these fixes.
The fix for the malformed query URN issue was introduced in CLI version 1.5.0. Versions prior to this (e.g., 1.3.x, 1.4.x) may still exhibit the URL-encoding bug.