Skip to main content

Managing Large Volume Hive Partition Ingestion

Angelo Bresci
Updated

Area: Ingestion Issues

Sub-Area: Hive Ingestion

Issue

DataHub Hive ingestion can become extremely slow or fail when processing tables with a large number of partitions. This occurs because the default behavior attempts to ingest metadata for all partitions, which can overwhelm the system when dealing with tables containing thousands or tens of thousands of partitions.

You Might Be Asking

  • Why is my Hive ingestion taking hours to complete or timing out?
  • How can I speed up ingestion for heavily partitioned Hive tables?
  • Can I limit the number of partitions DataHub ingests from Hive?

Solution

Configure partition ingestion limits in your Hive ingestion recipe to control the number of partitions processed per table:

  1. Add partition configuration to your Hive recipe: 
    source:
  type: hive
  config:
    metastore_uri: "thrift://:9083"
    partition_parallelism: 10
    max_partitions: 100
    partition_fetch_batch_size: 50
  2. For tables with extremely high partition counts, consider using selective ingestion patterns: 
    source:
  type: hive
  config:
    metastore_uri: "thrift://:9083"
    table_pattern:
      allow:
        - "database_name.table_name"
    partition_parallelism: 5
    max_partitions: 50
  3. Alternatively, disable partition ingestion entirely for performance-critical scenarios: 
    source:
  type: hive
  config:
    metastore_uri: "thrift://:9083"
    include_table_lineage: true
    include_column_lineage: false
    partition_parallelism: 0
  4. Monitor ingestion performance and adjust limits based on your infrastructure capacity: 
    # Check ingestion logs for partition processing times
kubectl logs  | grep "partition"

Additional Notes

Setting partition_parallelism to 0 disables partition metadata ingestion entirely, which significantly improves performance but removes partition-level lineage information. The max_partitions setting limits the total number of partitions ingested per table, while partition_fetch_batch_size controls how many partitions are fetched in each batch from the Hive metastore. Start with conservative values and increase based on your system's performance.

Related Documentation

Tags: hive, partitions, ingestion, performance, metastore, parallelism, batch-size, timeout, optimization, large-tables