Databricks Hive Metastore Ingestion Without Unity Catalog: Handling Workspaces Without a Metastore – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Databricks / Hive Metastore Source Configuration

Issue

When ingesting metadata from a Databricks Hive Metastore into DataHub using the legacy hive source with the databricks+pyhive scheme, ingestion fails whenever the underlying Databricks cluster or SQL Warehouse is inactive at the scheduled execution time. Users who attempt to migrate to the recommended unity-catalog connector as an alternative encounter a secondary problem: if the Databricks workspace has no Unity Catalog metastore assigned, the connector immediately fails with a "No metastore assigned" error before it can read any Hive Metastore tables — regardless of catalog or table pattern configuration. This leaves teams stuck between a compute-dependent legacy connector and a UC connector that cannot operate without a metastore assignment.

Error Messages

No metastore assigned for the current workspace
Connection error: cluster is inactive or terminated

You Might Be Asking

Can I use the unity-catalog source to ingest only Hive Metastore tables if my workspace has no UC metastore?
Is there a way to ingest Databricks Hive Metastore metadata without requiring a running cluster at ingestion time?
Why does the unity-catalog connector fail even when I set include_hive_metastore: true and filter to only hive_metastore catalogs?
What is the difference between the unity-catalog source and the legacy hive source for Databricks?

Solution

There are two distinct resolution paths depending on whether Unity Catalog can be enabled on your Databricks workspace.

Understanding the Root Cause

The unity-catalog DataHub source always makes an initial call to the Databricks Unity Catalog REST API (to enumerate catalogs and metastores) before processing any tables — including legacy Hive Metastore tables. If no UC metastore is assigned to the workspace, this first API call fails and the connector aborts entirely. There is no recipe-level workaround to bypass this behavior. A metastore assignment is a hard prerequisite for the unity-catalog connector.

Option 1 (Recommended): Assign a Unity Catalog Metastore and Use the `unity-catalog` Source

Have a Databricks account admin assign a Unity Catalog metastore to the workspace via the Databricks Account Console.
Create a Serverless SQL Warehouse in Databricks (SQL Warehouses → Create → Type: Serverless). Serverless warehouses auto-start on demand (typically within ~5 seconds) and stop automatically when idle, eliminating the cold-start compute dependency entirely.
Grant the service principal or personal access token the following minimum permissions:
- USE CATALOG on target catalogs
- USE SCHEMA on target schemas
- SELECT on system.information_schema (required for Hive Metastore metadata)

Configure your DataHub ingestion recipe using the unity-catalog source with include_hive_metastore: true:

source:
  type: unity-catalog
  config:
    workspace_url: https://<your-workspace>.azuredatabricks.net
    token: <your-databricks-personal-access-token>
    warehouse_id: <your-serverless-sql-warehouse-id>

    # Include legacy Hive Metastore tables alongside UC objects
    include_hive_metastore: true

    # Optionally restrict to only Hive Metastore catalog
    catalogs:
      - hive_metastore

    # Optional: restrict to specific schemas and tables
    schema_pattern:
      allow:
        - ".*"
    table_pattern:
      allow:
        - ".*"

sink:
  type: datahub-rest
  config:
    server: https://<your-datahub-instance>/gms
    token: <your-datahub-access-token>

Schedule the ingestion in DataHub. Because the Serverless warehouse starts automatically on the first API call, no compute needs to be pre-warmed before the scheduled run time.

Option 2 (Legacy / Maintenance Mode): Use the `hive` Source with JDBC

If Unity Catalog cannot be enabled on the workspace, you can continue using DataHub's legacy hive source, which connects directly via HiveServer2/JDBC and bypasses the Databricks UC API entirely. Note that this connector is in maintenance mode — it receives only critical bug fixes and may not receive new feature development. Use this path only if UC enablement is not feasible.

Switch your Databricks SQL Warehouse from Classic to Serverless type to eliminate the compute cold-start problem. Serverless warehouses auto-start on the incoming JDBC connection and do not need to be manually running at the scheduled ingestion time.

Configure your DataHub ingestion recipe using the hive source type:

source:
  type: hive
  config:
    scheme: databricks+pyhive
    host_port: <your-workspace>.azuredatabricks.net:443
    database: <your-database-or-schema>
    username: token
    password: <your-databricks-personal-access-token>
    options:
      connect_args:
        http_path: /sql/protocolv1/o/<workspace-id>/<cluster-or-warehouse-id>

sink:
  type: datahub-rest
  config:
    server: https://<your-datahub-instance>/gms
    token: <your-datahub-access-token>

Update the http_path to point to a Serverless SQL Warehouse rather than a Classic cluster. The path format for SQL Warehouses is typically /sql/1.0/warehouses/<warehouse-id>.

Feature Comparison

unity-catalog source: Uses Databricks REST API for core metadata (catalogs, schemas, tables, columns, views, lineage). Most metadata extraction requires zero active compute. A warehouse is required only for tag extraction, Hive Metastore tables, system-table lineage/usage, and profiling. Actively maintained and recommended.
hive source (JDBC): Requires an active HiveServer2 endpoint (cluster or warehouse) for all metadata operations. Currently in maintenance mode — not recommended for new deployments.

Additional Notes

The unity-catalog connector's include_hive_metastore: true flag is specifically designed to ingest legacy Hive Metastore tables alongside Unity Catalog-managed objects. It is not limited to UC-native tables. However, a UC metastore assignment is a hard prerequisite — there is no recipe configuration that bypasses the initial UC API handshake. If you are on a Classic SQL Warehouse and experiencing cold-start failures with the hive source, switching to a Serverless warehouse resolves the compute dependency for both connector types. Serverless warehouses incur no cost when idle and typically become available within seconds of an inbound connection. Plan for the administrative overhead of metastore assignment if your organization does not currently use Unity Catalog natively.