Ingestion Performance Tuning and Optimization

Large-scale ingestion jobs are taking too long, consuming excessive resources, or timing out. Organizations need strategies to optimize ingestion performance for large catalogs.

You Might Be Asking:

• How do I speed up ingestion for large catalogs?
• What's causing my ingestion to be slow?
• Can I parallelize ingestion?

1. Enable stateful ingestion for incremental updates:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    # Only ingest changes since last run
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

      # State backend
      state_provider:
        type: "datahub"
        config:
          datahub_api:
            server: "http://localhost:8080"

2. Optimize with parallel processing:

source:
  config:
    # Enable parallelism
    parallelism: 8  # Number of parallel threads

    # Batch processing
    batch_size: 100

    # Connection pooling
    max_workers: 10

3. Reduce ingestion scope:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    # Filter aggressively
    database_pattern:
      allow: ["PROD_DB", "ANALYTICS_DB"]
      deny: [".*_TEMP$", ".*_BACKUP$"]

    schema_pattern:
      allow: ["PUBLIC", "ANALYTICS.*"]
      deny: [".*_STAGING$"]

    table_pattern:
      deny: [".*_TEMP$", "TMP_.*"]

    # Disable expensive features
    include_table_lineage: false
    include_usage_stats: false
    profiling:
      enabled: false

4. Split large ingestions:

Example for Snowflake split by database. This can be adapted for other data sources/connectors.

# Split by database/schema into separate recipes
# Recipe 1: Production database
---
source:
  type: snowflake
  config:
    database_pattern:
      allow: ["PROD_DB"]
    platform_instance: "prod_snowflake"

# Recipe 2: Analytics database
---
source:
  type: snowflake
  config:
    database_pattern:
      allow: ["ANALYTICS_DB"]
    platform_instance: "prod_snowflake"

# Run in parallel or staggered

5. Optimize query-based features:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  config:
    # Optimize usage stats extraction
    include_usage_stats: true
    start_time: "-7d"  # Reduce from default 30d
    end_time: "-1d"

    # Optimize lineage extraction
    include_table_lineage: true
    sql_parser: "sqlglot"  # Faster than alternatives

    # Reduce query log sampling
    usage:
      top_n_queries: 5  # Reduce from 10

    # Use result caching
    use_queries_cache: true

6. Monitor and debug performance:

# Run with performance profiling
datahub ingest -c recipe.yml --performance-profiling

# Check execution time per component
cat profiling_results.json | jq '.timing'

# Identify bottlenecks
# Common slow operations:
# - Profiling large tables
# - Query history extraction
# - Schema metadata retrieval
# - Lineage extraction

# Use dry-run to test without emission
datahub ingest -c recipe.yml --dry-run

# Monitor resource usage during ingestion
# CPU, Memory, Network I/O
htop
iostat -x 1

Ingestion performance depends on source system performance, network latency, and DataHub capacity. For very large catalogs (100K+ entities), consider sharding ingestion across multiple workers. Monitor source system load to avoid impacting production.

• Performance Guide
• Stateful Ingestion