Snowflake Ingestion OOM Failures: Memory Tuning and Recipe Optimization for Large Accounts – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Snowflake Connector — Memory Pressure, Recipe Configuration, Pod Eviction

Issue

Snowflake ingestion jobs on large accounts can fail with worker pod eviction or Out-of-Memory (OOM) kills before completing. This occurs because the Snowflake connector is one of DataHub's most memory-intensive sources. Large accounts with hundreds of thousands of tables, views, and high query volumes can exhaust the executor coordinator's memory budget — particularly when view lineage parsing, usage aggregation, and schema extraction all run concurrently in a single recipe. Concurrent ingestion of multiple sources (e.g., Snowflake and another BI connector) sharing the same executor compounds the problem further.

Error Messages

aborted due to worker pod eviction, crash, or restart
permission-error: no tables/views/streams found
OOMKilled (Kubernetes pod termination reason)

You Might Be Asking

Why does my Snowflake ingestion keep getting killed partway through, often during view parsing?
Why does ingestion succeed in one environment but fail in another?
How do I reduce Snowflake ingestion memory usage without losing lineage or schema data?
Why does my usage-only Snowflake recipe show a FAILURE status even though usage data was ingested?
How do I split a Snowflake recipe into metadata and usage pipelines?

Solution

Step 1 — Understand the Memory Hotspots

The Snowflake connector consumes large amounts of memory for several reasons:

View lineage parsing: For every view, the connector parses the CREATE VIEW DDL through the SQL parser and holds a schema resolver for all upstream tables in memory simultaneously. With hundreds of thousands of views, this alone can exceed 5–6 GB.
Usage aggregation: Queries against ACCOUNT_USAGE use multi-level CTEs and ARRAY_UNIQUE_AGG() aggregations. For high-volume accounts, result sets can be enormous.
Schema extraction parallelism: By default, 20 concurrent workers each open separate DB connections and buffer metadata simultaneously.
Prefetch threads: The Snowflake client uses 10 concurrent prefetch threads for result set chunks, multiplying memory usage for large result sets.

Step 2 — Scope Your Schema and View Patterns

Reduce the number of objects ingested by adding explicit deny patterns. Use schema_pattern.deny to exclude scratch, temporary, and irrelevant schemas. Use view_pattern.deny to exclude views that match common scratch/temporary naming conventions.

source:
  type: snowflake
  config:
    schema_pattern:
      deny:
        - .*\.SCRATCH_.*
        - .*\.GOOGLE_DRIVE
        - .*\.TELEMETRY
        - .*\.STAGING
    view_pattern:
      deny:
        - .*\.scratch_.*
        - .*\..*_tmp_.*
        - .*\..*_staging_.*

Note: schema_pattern.deny filters both tables and views within the matched schemas. view_pattern.deny applies an additional filter on view names on top of schema filtering.

Step 3 — Split the Recipe into Separate Metadata and Usage Pipelines

A single Snowflake recipe performs three expensive operations simultaneously: metadata extraction, lineage computation, and usage aggregation. Splitting these into separate pipelines with unique pipeline_name values allows you to schedule them independently, reducing peak memory consumption and avoiding overlap.

Metadata + Lineage recipe (run daily or on schema-change cadence):

pipeline_name: <your-org>_snowflake_metadata

source:
  type: snowflake
  config:
    account_id: "<your-snowflake-account>"
    username: "<your-service-account>"
    password: "<your-password>"
    role: "<your-role>"
    warehouse: "<your-warehouse>"
    schema_pattern:
      deny:
        - .*\.SCRATCH_.*
    include_usage_stats: false
    include_operational_stats: false
    include_table_lineage: true
    include_view_lineage: true
    profiling:
      enabled: false
    stateful_ingestion:
      enabled: true

Usage-only recipe (run daily or every few days):

pipeline_name: <your-org>_snowflake_usage

source:
  type: snowflake
  config:
    account_id: "<your-snowflake-account>"
    username: "<your-service-account>"
    password: "<your-password>"
    role: "<your-role>"
    warehouse: "<your-warehouse>"
    include_usage_stats: true
    include_operational_stats: true
    include_table_lineage: false
    include_view_lineage: false
    include_tables: false
    include_views: false
    profiling:
      enabled: false
    stateful_ingestion:
      enabled: true

Important: Each pipeline must have a unique pipeline_name. This value is used by stateful ingestion to track what has been seen across runs (enabling soft deletes, incremental profiling, etc.). No two pipelines should share the same name.

Step 4 — Stagger Schedules to Avoid Concurrent Memory Pressure

If multiple ingestion sources share the same executor coordinator, running them concurrently will cause them to compete for memory and may result in both being OOM killed. Stagger your schedules so that:

The Snowflake metadata pipeline and the Snowflake usage pipeline do not overlap.
Other connector ingestions (e.g., BI tools like Mode, Looker, Tableau) are scheduled to run when Snowflake ingestion is not active.

Step 5 — Disable or Tune View Lineage if Memory Pressure Persists

If the ingestion still fails during the view parsing stage after scoping schemas and views, you can reduce memory further by disabling column-level lineage from view definitions. You will still get all views as datasets; only the SQL parse step is skipped.

source:
  type: snowflake
  config:
    include_view_lineage: false

Alternatively, keep include_view_lineage: true but ensure your view_pattern.deny is aggressive enough to exclude large scratch/staging view populations.

Step 6 — Fix False FAILURE Status on Usage-Only Recipes

A usage-only Snowflake recipe may show a FAILURE pipeline status even when usage data is successfully ingested. This happens because the source runs an object discovery step at startup to build a schema lookup map for query parsing. If the role used for usage ingestion has access to ACCOUNT_USAGE but not to individual table metadata (SELECT or REFERENCES), the discovery step finds zero datasets and emits a permission error, causing the pipeline to be marked as failed.

To suppress this and allow the pipeline to complete with a SUCCESS status, add:

source:
  type: snowflake
  config:
    warn_no_datasets: true

This converts the hard failure to a warning. The metadata pipeline remains authoritative for object discovery.

Step 7 — Enable Snowflake Tag Extraction

Tag extraction is disabled by default. To enable it, add the following to your metadata recipe:

source:
  type: snowflake
  config:
    extract_tags: without_lineage

Valid values are:

skip — No tags extracted (default)
without_lineage — Extract tags without propagating through lineage (recommended for most cases)
with_lineage — Extract and propagate tags through lineage

The service account role must have SELECT on SNOWFLAKE.ACCOUNT_USAGE.TAG_REFERENCES for this to function. Missing permissions will appear as warnings in the run report.

Step 8 — Optionally Promote Specific Tags as Structured Properties

If you want certain Snowflake tags to appear as structured properties in DataHub rather than regular tags, create a dedicated third pipeline. Use structured_property_pattern.allow to target only the specific tags you want promoted, and add those same tags to tag_pattern.deny in your main metadata recipe to avoid duplication.

Structured properties recipe:

pipeline_name: <your-org>_snowflake_structured_tags

source:
  type: snowflake
  config:
    account_id: "<your-snowflake-account>"
    username: "<your-service-account>"
    password: "<your-password>"
    role: "<your-role>"
    warehouse: "<your-warehouse>"
    extract_tags: without_lineage
    extract_tags_as_structured_properties: true
    structured_property_pattern:
      allow:
        - <DATABASE>\.<SCHEMA>\.<tag_name_1>
        - <DATABASE>\.<SCHEMA>\.<tag_name_2>
    include_tables: false
    include_views: false
    include_table_lineage: false
    include_view_lineage: false
    profiling:
      enabled: false
    stateful_ingestion:
      enabled: true

In your main metadata recipe, exclude those same tags:

source:
  type: snowflake
  config:
    extract_tags: without_lineage
    tag_pattern:
      deny:
        - <DATABASE>\.<SCHEMA>\.<tag_name_1>
        - <DATABASE>\.<SCHEMA>\.<tag_name_2>

The fully qualified tag pattern format is <database>.<schema>.<tag_name>.

Additional Notes

For DataHub Cloud (managed) deployments, customers cannot directly adjust pod memory limits. If your account has a large schema footprint and recipe-level optimizations alone are insufficient, contact DataHub Support to request executor memory scaling for your environment.
Memory requirements vary significantly by account size. Environments with 300K+ tables or 200K+ views will require substantially more memory than the default executor allocation. POC/staging environments may have been provisioned with different memory limits than production — confirm that both environments are sized equivalently.
Table profiling (column-level statistics such as min, max, distinct count) is separate from table-level statistics (row count, size). If the ingestion role does not have SELECT on table data or access to INFORMATION_SCHEMA table stats, disable profiling entirely with profiling.enabled: false to avoid errors and reduce memory usage.
The extract_tags_as_structured_properties flag is global per recipe — you cannot mix regular tags and structured property tags within a single source run. Use the split-pipeline pattern described in Step 8 for mixed tag handling.
ING-2699 (debug log explosion from make_curl_command) was an active issue at the time this guidance was developed. If you are running with verbose/debug logging enabled, this may contribute to additional memory pressure from log buffering. Ensure log levels are set appropriately for production ingestion runs.