dbt Ingestion Stateful Ingestion and Pipeline Name Best Practices to Avoid Excessive Soft-Deletions – DataHub Customer Support Portal

```html

Area: Ingestion Issues

Sub-Area: dbt Ingestion / Stateful Ingestion Configuration

Issue

When multiple dbt ingestion pipelines share the same pipeline name in DataHub's stateful ingestion configuration, the platform can generate an unexpectedly high volume of metadata change proposals (MCPs). This occurs because stateful ingestion uses the pipeline name as a key to track previously seen entities; when pipelines with overlapping scopes share a name, DataHub repeatedly issues soft-deletion events for entities it believes have disappeared between runs. Over time, this can produce a sustained ingestion spike that exceeds the platform's processing capacity, causes Kafka consumer lag to grow into the tens of millions of messages, and ultimately triggers rate-limiting (HTTP 429) errors that block downstream pipelines and production workflows.

Error Messages

'Unable to emit metadata to DataHub GMS: None', {'error': 'Rate limit exceeded'}
HTTPError: 429 Client Error: Too Many Requests for url: https://<your-instance>.acryl.io/api/graphql

You Might Be Asking

Why are my dbt ingestion pipelines suddenly producing far more events than they used to?
Why does stateful ingestion keep soft-deleting entities that still exist in my dbt project?
Why am I receiving HTTP 429 rate-limit errors from DataHub even though I have not changed my ingestion volume?
How should I name my DataHub ingestion pipelines to avoid conflicts with stateful ingestion?
How can I monitor Kafka consumer lag in DataHub Cloud to detect a growing backlog early?

Solution

1. Assign a Unique Pipeline Name to Every Ingestion Pipeline

The most critical fix is to ensure each ingestion pipeline has a globally unique pipeline_name. DataHub's stateful ingestion checkpointing uses this name to store and compare the set of entities seen in previous runs. If two or more pipelines share the same name, they will overwrite each other's checkpoints, causing each run to perceive large numbers of entities as newly removed and emit unnecessary soft-deletion MCPs.

Open each dbt ingestion recipe file (YAML or JSON) and locate the top-level pipeline_name field.
Set a name that uniquely identifies both the source and the logical scope of the pipeline, for example:
```
pipeline_name: dbt_production_warehouse_core
```
or for multiple environments:
```
pipeline_name: dbt_staging_marketing
```
Never reuse the same pipeline_name across two recipes that ingest different dbt projects, profiles, or node subsets.

2. Verify Stateful Ingestion Is Enabled and Correctly Scoped

Confirm that stateful ingestion settings are explicitly defined and scoped to match the pipeline's actual coverage. A minimal correct configuration looks like the following:

source:
  type: dbt
  config:
    project_name: <your-dbt-project-name>
    manifest_path: <path-to-manifest.json>
    catalog_path: <path-to-catalog.json>
    run_results_path: <path-to-run-results.json>
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

pipeline_name: dbt_<environment>_<project>_<unique-scope>

sink:
  type: datahub-rest
  config:
    server: https://<your-instance>.acryl.io/api/gms
    token: <your-datahub-token>

3. Stagger and Deduplicate Ingestion Schedules

Even correctly named pipelines can overload the platform if many run concurrently. Apply the following scheduling best practices:

Offset pipeline cron schedules by at least 5–10 minutes so they do not all emit MCPs simultaneously.
Avoid triggering the same logical pipeline more than once in a short window (e.g., from both a scheduler and a manual run).

Reduce parallelism settings if your dbt project is very large:

source:
  config:
    # Reduce concurrent API calls to the dbt source
    max_threads: 5

4. Implement Retry Logic with Exponential Backoff

If your ingestion framework makes direct REST or GraphQL calls, add retry logic so that transient 429 errors do not immediately fail the pipeline. Example using the DataHub Python SDK:

import time
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter(
    gms_server="https://<your-instance>.acryl.io/api/gms",
    token="<your-datahub-token>",
    retry_max_times=5,          # retry up to 5 times
    retry_status_codes=[429, 503],
)

When implementing custom retry loops, use exponential backoff:

import time

def emit_with_backoff(emitter, mcp, max_retries=5):
    for attempt in range(max_retries):
        try:
            emitter.emit(mcp)
            return
        except Exception as e:
            if "429" in str(e) or "Rate limit" in str(e):
                wait = 2 ** attempt  # 1, 2, 4, 8, 16 seconds
                print(f"Rate limited. Retrying in {wait}s (attempt {attempt + 1})")
                time.sleep(wait)
            else:
                raise
    raise RuntimeError("Max retries exceeded due to rate limiting")

5. Monitor Kafka Consumer Lag via the DataHub Cloud Operations API

DataHub Cloud exposes an endpoint to check the MCP consumer lag. Monitoring this proactively allows you to detect a growing backlog before it causes user-visible failures. Use the following request (requires a valid DataHub token):

curl -X GET \
  'https://<your-instance>.acryl.io/openapi/operations/messaging/mcp/consumer/lag?skipCache=false&detailed=true' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <your-datahub-token>'

Example response structure (values are illustrative):

{
  "transport": "kafka",
  "consumerGroups": {
    "<consumer-group-id>": {
      "<topic-name>": {
        "partitions": {
          "0": { "offset": 1000000, "lag": 50000 },
          "1": { "offset": 1000000, "lag": 0 }
        },
        "metrics": {
          "maxLag": 50000,
          "medianLag": 0,
          "totalLag": 50000,
          "avgLag": 25000
        }
      }
    }
  }
}

Alert on totalLag exceeding a threshold appropriate for your ingestion volume (e.g., 500,000 messages). A rapidly growing lag combined with a high soft-deletion rate in dbt ingestion logs is a strong indicator of the shared-pipeline-name problem described above.

6. Audit Existing Pipeline Names and Rename Conflicting Pipelines

List all ingestion sources configured in DataHub (UI: Ingestion → Sources).
Identify any two sources of type dbt that share the same pipeline_name.
Rename one or both pipelines following the unique-naming convention in Step 1.
After renaming, the old checkpoint key will no longer exist, so the first run under the new name will treat all entities as new. This is expected and will not cause data loss — it simply means no soft-deletions will be emitted on that first run, and the checkpoint will be freshly established.

Additional Notes

Stateful ingestion pipeline name conflicts are a silent misconfiguration — DataHub does not raise an explicit error when two pipelines share a name. The symptom (excessive soft-deletions and MCP volume) may only become visible days or weeks after the configuration is introduced, particularly as the dbt project grows or ingestion frequency increases. The remove_stale_metadata flag amplifies the impact: while it is useful for keeping the catalog clean, it should only be enabled when the pipeline scope is stable and uniquely named. This behavior applies to any DataHub ingestion source that supports stateful ingestion, not only dbt, but dbt pipelines are particularly susceptible because they tend to cover large numbers of nodes. In DataHub Cloud (Acryl-managed), sustained MCP backlogs may trigger platform-level protective rate limits (HTTP 429); these limits exist to protect platform stability for all tenants and are not configurable by end users. If your ingestion volume consistently exceeds platform thresholds after following all best practices above, contact DataHub Support to discuss throughput options.