Snowflake Ingestion Performance Issues

Snowflake ingestion takes an extremely long time to complete, times out, or consumes excessive resources. This commonly occurs with large Snowflake deployments due to querying expensive metadata views, lack of query result caching, or attempting to ingest all historical query logs.

You Might Be Asking:

• Why is my Snowflake ingestion so slow?
• How can I speed up Snowflake metadata extraction?
• Can I reduce the cost of Snowflake ingestion?

1. Optimize ingestion configuration:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    # Limit the scope
    schema_pattern:
      allow: ["^analytics\..*"]
      deny: [".*_backup$"]

    # Reduce usage/lineage lookback
    include_usage_stats: true
    start_time: "-7d"  # Only last 7 days instead of default 30

    # Disable expensive features if not needed
    include_table_lineage: true
    include_view_lineage: false  # Disable if not required
    include_column_lineage: false  # Very expensive

    # Use profiling sparingly
    profiling:
      enabled: false  # Enable only for specific schemas

2. Use a dedicated warehouse for ingestion:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  config:
    warehouse: "DATAHUB_INGESTION_WH"
    # Use a smaller warehouse
    # Run ingestion during off-peak hours

3. Enable result caching in Snowflake:

Example for Snowflake. This can be adapted for other data sources/connectors.

ALTER WAREHOUSE DATAHUB_INGESTION_WH SET 
  ENABLE_QUERY_ACCELERATION = FALSE
  AUTO_RESUME = TRUE
  AUTO_SUSPEND = 60;

-- Ensure result cache is enabled (default)
ALTER SESSION SET USE_CACHED_RESULT = TRUE;

4. Use incremental ingestion with stateful mode:

source:
  config:
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

5. Parallelize ingestion for large deployments:
   • Split into multiple recipes by schema/database
   • Use different platform_instance values
   • Schedule them with staggered start times

Monitor Snowflake query history to identify expensive DataHub queries. The accountusage views can have significant latency (up to 120 minutes), consider using informationschema for more real-time data.

• Snowflake Source Configuration
• Performance Tuning Guide