Hive Metastore Connector: Use database_pattern Instead of schema_pattern for Filtering – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Hive Metastore Connector Configuration

Issue

When configuring the Hive Metastore connector with schema_pattern allow and deny filters, the filters are silently ignored and have no effect on ingestion. All Hive databases are ingested regardless of the allow or deny patterns specified under schema_pattern. This behavior can be confusing because schema_pattern works correctly in other SQL-based connectors, and older versions of the Hive Metastore documentation contained examples that showed schema_pattern as a valid filter option for this connector.

Error Messages

No error is raised. The connector silently accepts schema_pattern but does not apply it.

You Might Be Asking

Why does my schema_pattern allow list have no effect in the Hive Metastore connector?
Why are schemas I added to the schema_pattern.deny list still being ingested?
Why does schema_pattern work in other connectors but not in Hive Metastore?
How do I filter Hive databases or schemas during ingestion with the hive-metastore source?

Solution

Understand why schema_pattern does not work for the Hive Metastore connector. In Hive, a "database" and a "schema" refer to the same namespace level — CREATE SCHEMA and CREATE DATABASE are interchangeable. However, the hive-metastore connector internally refers to this level as a database, not a schema. All filtering logic in the connector (in the metadata processor and the Thrift fetcher) uses database_pattern. The schema_pattern field is inherited from a shared SQL config base class and is accepted without error, but it is never read or applied during ingestion.

Locate the schema_pattern block in your Hive Metastore ingestion recipe. It will look similar to the following:

source:
  type: hive-metastore
  config:
    # ... other config ...
    schema_pattern:
      allow:
        - "^my_database_1$"
        - "^my_database_2$"
      deny:
        - "land_.*"
        - "ws_.*"

Replace schema_pattern with database_pattern. The structure of the allow and deny lists remains identical:

source:
  type: hive-metastore
  config:
    # ... other config ...
    database_pattern:
      allow:
        - "^my_database_1$"
        - "^my_database_2$"
      deny:
        - "land_.*"
        - "ws_.*"

Remove any remaining schema_pattern block entirely from the recipe. Do not set both database_pattern and schema_pattern at the same time, as schema_pattern will continue to be ignored and may cause confusion.
Run the ingestion pipeline and verify the results. Check the ingestion report to confirm that databases matching the deny patterns appear as filtered and that the total container count has decreased accordingly.
```
# Example: run via CLI
datahub ingest -c my_hive_metastore_recipe.yml
```
If you have stateful ingestion with stale-entity removal enabled, the next successful run after applying database_pattern will automatically soft-delete any previously ingested databases that are now excluded by the filter.

Additional Notes

Allow/Deny pattern behavior: AllowDenyPattern matching is case-insensitive by default, and deny rules take priority over allow rules. If a database name matches both an allow pattern and a deny pattern, it will be excluded.

Anchored patterns: When using exact database names in the allow list, anchoring the pattern (e.g., ^my_database$) ensures that only that exact name matches and prevents partial matches.

Unrelated settings: The database and metastore_db_name configuration options in the Hive Metastore connector refer to the backend metastore database (e.g., the MySQL or PostgreSQL instance storing Hive metadata). These are distinct from database_pattern and do not control which Hive schemas are filtered during ingestion.

Documentation correction: Earlier versions of the Hive Metastore connector documentation and example recipes incorrectly showed schema_pattern as a valid filter. This has been corrected in the official documentation to use database_pattern.

Other connectors: This behavior is specific to the hive-metastore source type. Other SQL-based connectors (e.g., hive, trino, mysql) may correctly honor schema_pattern. Do not assume filter parameter names are interchangeable across connector types.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation