Snowflake Ingestion Performance: include_views and lazy_schema_resolver Interaction – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Snowflake Connector Configuration & Performance

Issue

When setting include_views: false in a Snowflake ingestion recipe, users may observe that ingestion takes significantly longer than when include_views: true is set — even when no views are targeted by the recipe. This is counterintuitive because excluding views is expected to reduce the scope of work. The root cause is an interaction between include_views: false and lazy_schema_resolver: false: together, these two settings trigger an eager, bulk schema fetch from the DataHub API that operates at the full platform-and-environment scope rather than at the scope defined by your database_pattern, schema_pattern, and table_pattern filters. In large Snowflake environments with hundreds of thousands of datasets, this bulk fetch can add tens of minutes of overhead to every ingestion run.

Error Messages

INFO {datahub.sql_parsing.schema_resolver_provider:69} - Fetching schemas for platform snowflake, env <ENV>
INFO {datahub.sql_parsing.schema_resolver_provider:90} - Finished loading <N> schema info in <seconds> seconds

You Might Be Asking

Why does setting include_views: false make my Snowflake ingestion slower instead of faster?
Why is DataHub loading hundreds of thousands of schemas when my recipe only targets a few hundred tables?
What does lazy_schema_resolver do, and should I set it to true or false?
How do I stop the "Fetching schemas for platform snowflake" bulk load from running during ingestion?

Solution

Understand the root cause.
When include_views: false is set, the connector interprets this as "schema metadata will not be populated incrementally during ingestion via views." If lazy_schema_resolver is also set to false, the connector responds by eagerly pre-loading all schema information for the entire Snowflake platform and environment from the DataHub API before ingestion begins. This bulk fetch is scoped to the platform (e.g., snowflake) and environment (e.g., DEV or PROD) — not to your database_pattern, schema_pattern, or table_pattern filters. As a result, all schemas registered in DataHub for that platform/environment are loaded, regardless of how narrowly your recipe is scoped.
Set lazy_schema_resolver: true in your recipe.
This is the primary fix. With lazy_schema_resolver: true, schemas are fetched on demand only for the tables actually being processed, eliminating the bulk pre-load. Note that true is the default value for this setting; if it was not explicitly set in your recipe, it may have been set to false elsewhere or inherited from a shared configuration.
```
source:
  type: snowflake
  config:
    include_views: false
    lazy_schema_resolver: true  # Set to true (this is also the default)
```
Verify the bulk schema load no longer occurs.
After applying the change, re-run your ingestion. The following log lines should no longer appear:
```
INFO {datahub.sql_parsing.schema_resolver_provider:69} - Fetching schemas for platform snowflake, env <ENV>
INFO {datahub.sql_parsing.schema_resolver_provider:90} - Finished loading <N> schema info in <seconds> seconds
```
If those lines are absent, the eager bulk fetch has been successfully suppressed and schemas are now being resolved lazily on demand.
(Optional) Further restrict recipe patterns for large environments.
As a complementary best practice for large Snowflake deployments, narrow your database_pattern, schema_pattern, and table_pattern filters as tightly as possible to minimize the number of objects the connector evaluates. In environments with hundreds of thousands of datasets, splitting ingestion into multiple smaller recipes targeting specific schemas or databases can also improve reliability and performance.
```
source:
  type: snowflake
  config:
    include_views: false
    lazy_schema_resolver: true
    database_pattern:
      allow:
        - "^MY_DATABASE$"
    schema_pattern:
      allow:
        - "^MY_SCHEMA$"
    table_pattern:
      allow:
        - "^MY_TABLE_PREFIX.*"
```

Additional Notes

The default value of lazy_schema_resolver is true. If you did not explicitly add lazy_schema_resolver: false to your recipe but are still seeing the bulk schema fetch behavior, check for shared or inherited base configurations that may be setting it. The bulk fetch triggered by lazy_schema_resolver: false combined with include_views: false is not bounded by any pattern filters — it loads every schema registered under the given platform and environment in DataHub. In large environments (100k+ datasets), this can add 30–60 minutes or more of overhead per ingestion run. This behavior is specific to the interaction of these two settings; using include_views: false alone with the default lazy_schema_resolver: true does not trigger the bulk fetch.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation