Snowflake Ingestion Failing Due to Incorrect Allowlist or Denylist Regex Patterns – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Snowflake Source Configuration — Pattern Filtering

Issue

Snowflake ingestion runs fail or produce no results (zero tables and views scanned) when schema_pattern, table_pattern, or other pattern-based filter fields in the recipe contain invalid regular expressions. A common mistake is writing patterns in glob syntax — for example, using a leading * instead of the regex equivalent .*. Because DataHub's pattern fields expect standard Python regular expressions, a glob-style leading wildcard causes a re.error at runtime. Depending on where the invalid pattern is evaluated, the error may be silently downgraded to a warning during schema extraction (causing no schemas to be ingested) while simultaneously terminating the run during query history extraction. Additional subtle issues — such as YAML quoting mistakes that corrupt a pattern or using fully qualified names when match_fully_qualified_names is false — can cause filters to silently match nothing without raising an error.

Error Messages

<class 're.error'>: nothing to repeat at position 0
Failed to get schemas for database <DATABASE_NAME>
No schemas found in database. If schemas exist, please grant USAGE permissions on them.

You Might Be Asking

Why does my Snowflake ingestion show tables_scanned: 0 and views_scanned: 0 even though I know tables exist?
Why does the ingestion warn about missing USAGE permissions when permissions are correctly granted?
Why does my denylist appear to be blocking entries that I have explicitly added to my allowlist?
Why does a pattern that looks correct in my recipe not actually exclude the schemas I expect?

Solution

Use regular expressions, not glob syntax, in all pattern fields.
DataHub recipe pattern fields (schema_pattern, table_pattern, database_pattern, etc.) accept Python regular expressions. Replace any leading glob wildcard * with the regex equivalent .*.

Invalid (glob) Correct (regex)

*SANDBOX .*SANDBOX

*STAGING.*_SENSITIVE .*STAGING.*_SENSITIVE
Ensure denylist patterns do not accidentally suppress allowlist entries.
The denylist always takes precedence over the allowlist. If you want to deny a broad pattern but permit a specific sub-pattern, use a negative lookahead inside the deny pattern itself.

Example: To deny all schemas containing STAGING but allow schemas that begin with DBT_STAGING_:
```
schema_pattern:
  allow:
    - 'DBT_STAGING_.*'
  deny:
    - '(?!.*DBT_STAGING_).*STAGING.*'
```
Without the negative lookahead, the deny entry .*STAGING.* would match and block DBT_STAGING_* schemas even though they appear in the allowlist.
Verify YAML quoting does not corrupt your regex patterns.
Trailing or mismatched quotes in YAML can silently alter a pattern. For example, a pattern ending in ''' in YAML is interpreted as an escaped apostrophe appended to the pattern string, so (?!.*DBT_STAGING_).*STAGING.*' would only match names containing a literal apostrophe after STAGING — which schema names never contain. Always inspect the final resolved pattern string, not just the YAML source.
```
# Incorrect — trailing apostrophe corrupts the pattern
deny:
  - '(?!.*DBT_STAGING_).*STAGING.*'''

# Correct
deny:
  - '(?!.*DBT_STAGING_).*STAGING.*'
```
Match the pattern scope to the match_fully_qualified_names setting.
By default, match_fully_qualified_names is false, meaning schema_pattern is compared against the schema name alone, not database.schema. A pattern like PROD.DBT_TEST__AUDIT will never match in this mode because schema names do not include the database prefix.
- Option A (recommended for most cases): Use only the schema name in the pattern.
```
schema_pattern:
  deny:
    - 'DBT_TEST__AUDIT'
```
- Option B: Enable fully qualified name matching and use the qualified form. Note that . is a regex wildcard and must be escaped to match a literal dot.
```
source:
  config:
    match_fully_qualified_names: true
    schema_pattern:
      deny:
        - 'PROD\.DBT_TEST__AUDIT'
```

Reference: corrected schema_pattern deny block example.

The following illustrates a fully corrected deny list that avoids all of the issues described above:

schema_pattern:
  deny:
    - 'DBT_TEST__AUDIT'
    - '.*SANDBOX_RESTRICTED_CUSTOMER'
    - '.*SANDBOX_RESTRICTED_EXAMPLE'
    - '.*SANDBOX_SENSITIVE'
    - '.*SANDBOX'
    - '.*DEMO_RESTRICTED_CUSTOMER'
    - '.*DEMO_RESTRICTED_EXAMPLE'
    - '.*DEMO_SENSITIVE'
    - '.*DEMO'
    - '.*STAGING.*_RESTRICTED_CUSTOMER'
    - '.*STAGING.*_RESTRICTED_EXAMPLE'
    - '.*STAGING.*_SENSITIVE'
    - '(?!.*DBT_STAGING_).*STAGING.*'

Do not misinterpret the permissions warning.
When an invalid pattern causes schema fetching to fail, the ingestion log may display: "No schemas found in database. If schemas exist, please grant USAGE permissions on them." This message is misleading in this context. The schema fetch failed because the regex pattern could not be compiled, not because of a missing permission. Fix the pattern before investigating Snowflake permission grants.

Additional Notes

Pattern field validation happens at runtime, not at recipe save time in the DataHub UI, so an invalid pattern may not be flagged until the ingestion actually runs. During schema extraction, pattern compilation errors are caught and downgraded to warnings, which can make the failure appear to be a permissions or connectivity issue rather than a configuration problem. However, the same invalid pattern used in query history extraction (when use_queries_v2 is enabled) is not wrapped in the same error handler and will terminate that phase of the run. This means a single misconfigured pattern can produce two different failure modes in the same run. Always test regex patterns independently (e.g., using Python's re module) before adding them to a recipe.

Invalid (glob)	Correct (regex)
`*SANDBOX`	`.*SANDBOX`
`STAGING._SENSITIVE`	`.STAGING._SENSITIVE`

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation