Skipping MLflow ML Model Ingestion in Databricks to Speed Up Ingestion – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Databricks / Unity Catalog Connector Configuration

Issue

When running Databricks ingestion in DataHub, the connector automatically processes all MLflow registered models and their versions by default. In environments with large model registries — for example, a model group with hundreds of child model versions — this can significantly slow down ingestion runs. Each model version triggers multiple individual API calls (listing model versions, fetching run details, and reading model signature files), resulting in potentially thousands of API calls and drastically increased ingestion time. Users who do not need ML model metadata ingested into DataHub have no straightforward way to skip this processing in older versions of the CLI. The ml_model_max_results field was found to be ineffective at preventing ML model ingestion even when set to 0 or 1.

Error Messages

sink wrote workunit urn:li:mlModel:(urn:li:dataPlatform:databricks,<catalog>.<schema>.<model_name>,PROD)-status — repeated hundreds or thousands of times in debug logs, indicating ML models are still being ingested despite configuration attempts to suppress them.

You Might Be Asking

How do I skip MLflow model ingestion in the Databricks connector to speed up my ingestion pipeline?
Why is ml_model_max_results: 0 not preventing ML model ingestion in Databricks?
Is there a flag to completely disable ML model or MLflow experiment processing in the DataHub Databricks source?
My Databricks ingestion is very slow — can I exclude registered models to reduce runtime?

Solution

Upgrade your DataHub ingestion CLI to version 1.7.0 or later.
The include_ml_models configuration flag was introduced in CLI version 1.7.0 (tracked in PR #18220). Earlier versions do not support this flag, and the previously suggested workaround of setting ml_model_max_results: 0 is not effective — it does not prevent the ML model ingestion pipeline from running.

If your DataHub deployment has not yet defaulted to CLI 1.7.0, you can pin the CLI version explicitly via the Advanced Settings menu in your ingestion source configuration in the DataHub UI.

Add include_ml_models: false to your Databricks ingestion recipe.

Update your ingestion recipe YAML to include the new flag under the config block:

source:
  type: databricks
  config:
    workspace_url: "https://<your-databricks-workspace>.cloud.databricks.com/"
    token: "${DATABRICKS_TOKEN}"
    warehouse_id: "<your-warehouse-id>"
    env: PROD
    include_table_lineage: true
    include_column_lineage: false
    include_ml_models: false
    catalog_pattern:
      allow:
        - "<your-catalog-name>"
    stateful_ingestion:
      enabled: true
      fail_safe_threshold: 90

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.acryl.io/gms"

Verify the fix.
After updating the recipe and CLI version, re-run the ingestion pipeline. Monitor the executor logs — you should no longer see repeated sink wrote workunit urn:li:mlModel:... entries. Total ingestion runtime should decrease substantially for environments with large model registries.
Note on CLI version pinning.
If you explicitly pin the CLI version to 1.7.0 in the Advanced Settings of your ingestion source, be aware that your ingestion will remain on that version until you manually update or clear the pin. In most managed DataHub deployments, the default CLI version is upgraded automatically alongside server upgrades. Once include_ml_models: false becomes a supported default behavior in a future release, you may be able to remove the explicit version pin.

Additional Notes

The ml_model_max_results field controls the page size for Databricks API calls that list registered models, but it does not gate whether the ML model ingestion pipeline runs at all. Setting it to 0 or 1 does not reliably skip ML model ingestion and is not a supported workaround. The only correct solution is to use include_ml_models: false with CLI 1.7.0 or later.

The performance impact of ML model ingestion scales with the number of registered model versions in your Databricks workspace. For each model version, the ingester makes multiple individual API calls — to list versions, fetch MLflow run details, and read model signature files — which can result in thousands of API calls for large model registries, causing significant ingestion slowdowns.

This flag was introduced as a fix in DataHub CLI 1.7.0 via GitHub PR #18220. It is expected to become the configurable default in DataHub server version 2.1.x and later.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation