Snowflake Assets Appearing in Mixed Case After Changing convert_urns_to_lowercase – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Snowflake Ingestion / URN Case Handling

Issue

When a Snowflake ingestion job is initially run without the convert_urns_to_lowercase configuration option (or with it set to false), assets such as databases, schemas, and tables are ingested with uppercase URNs. If the job fails partway through due to missing permissions, no clean stateful checkpoint is committed. After fixing the permissions and re-running with convert_urns_to_lowercase: true, new lowercase URNs are created successfully, but the previously ingested uppercase URNs are not removed even with stateful ingestion enabled. This results in Snowflake assets appearing in a mix of upper and lowercase in the DataHub catalog.

You Might Be Asking

Why didn't stateful ingestion soft-delete the old uppercase URNs when I re-ran with convert_urns_to_lowercase: true?
How do I clean up uppercase Snowflake URNs and ensure all assets are consistently lowercase?
Does changing convert_urns_to_lowercase after ingestion has already run cause duplicate assets?
Why are some Snowflake tables appearing under lowercase folders without a parent database container?

Root Cause

There are two compounding factors that cause this behaviour:

Failed first run prevents a clean checkpoint: Because the initial ingestion run failed (due to missing permissions), stateful ingestion never committed a complete checkpoint containing the uppercase URNs. When the second run succeeded, stateful ingestion compared the new lowercase URNs against an incomplete or empty prior checkpoint, so no orphan removal was triggered for the uppercase URNs.
URN key space change is not reconciled automatically: The convert_urns_to_lowercase setting is part of a dataset's URN identity. Changing it after ingestion has already run effectively re-keys every dataset. Stateful ingestion treats uppercase and lowercase URNs as completely different entities (exact string comparison), so it cannot automatically link old uppercase URNs to new lowercase ones for cleanup purposes. This is a documented limitation: "This value is part of the dataset's URN identity and is fixed for the life of a deployment. Changing it after ingestion has been run re-keys every dataset."

Additionally, if some tables appear only under lowercase folders with no parent database container, this typically indicates that another ingestion source (such as dbt Cloud) has registered those tables, but the Snowflake ingestion role does not have sufficient permissions to discover the corresponding schemas, so no database container is created for them.

Solution

Option 1 (Recommended): Soft-Delete All Snowflake Assets and Re-Run Ingestion

This is the cleanest approach. Soft-deleting all Snowflake assets is reversible, and re-running ingestion with convert_urns_to_lowercase: true will re-populate all assets consistently in lowercase.

Run the following DataHub CLI command to soft-delete all Snowflake assets:
```
datahub delete --platform snowflake --soft --force
```

Ensure your ingestion recipe has convert_urns_to_lowercase: true set explicitly:

source:
  type: snowflake
  config:
    convert_urns_to_lowercase: true
    # ... other config options

Re-run the Snowflake ingestion job. All assets will be re-ingested with lowercase URNs, and the uppercase URNs will remain soft-deleted.

Option 2 (Surgical): Target Only Uppercase URNs Using the DataHub Graph Client

If you need to preserve other Snowflake metadata and do not want to soft-delete everything, you can use the DataHub Python SDK to identify and soft-delete only the uppercase URNs.

Install the DataHub Python SDK if not already installed:
```
pip install acryl-datahub
```

Use the DataHubGraph client to search for and soft-delete uppercase Snowflake URNs. Example pattern:

from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph

graph = DataHubGraph(DatahubClientConfig(server="https://<your-instance>.acryl.io/gms"))

# Search for Snowflake datasets and filter those with uppercase URNs
for urn in graph.get_urns_by_filter(platform="snowflake", entity_types=["dataset"]):
    # Uppercase URNs will contain uppercase characters in the path segments
    if urn != urn.lower():
        print(f"Soft-deleting uppercase URN: {urn}")
        graph.soft_delete_entity(urn=urn)

After cleanup, re-run the Snowflake ingestion job with convert_urns_to_lowercase: true.

Preventing Checkpoint Issues on Subsequent Runs

To force stateful ingestion to establish a fresh checkpoint baseline after cleanup, add ignore_old_state: true for a single cleanup run only, then remove it:

source:
  type: snowflake
  config:
    convert_urns_to_lowercase: true
    # ... other config options

pipeline_name: snowflake_ingestion

stateful_ingestion:
  enabled: true
  ignore_old_state: true  # Remove this after the cleanup run!

Re-run ingestion once with this setting, then remove ignore_old_state: true and run again. Subsequent runs will correctly track the lowercase URN space.

Fixing Tables Without a Parent Database Container

If tables appear under lowercase folders with no parent database in the UI, this typically means the Snowflake ingestion service account role lacks access to those schemas. To resolve this:

Grant the DataHub reader role USAGE on the affected schemas and SELECT on their tables, including future tables. Example:

GRANT USAGE ON SCHEMA <your_database>.<your_schema> TO ROLE <datahub_reader_role>;
GRANT SELECT ON ALL TABLES IN SCHEMA <your_database>.<your_schema> TO ROLE <datahub_reader_role>;
GRANT SELECT ON FUTURE TABLES IN SCHEMA <your_database>.<your_schema> TO ROLE <datahub_reader_role>;

Re-run the Snowflake ingestion job. The previously orphaned tables should now appear under their correct database and schema containers.

Additional Notes

Default behaviour: For the Snowflake connector, convert_urns_to_lowercase defaults to true. If you are unsure whether your earlier runs used uppercase URNs, inspect the recipe attached to each ingestion run in the DataHub UI under Ingestion > Run History.
URN identity is immutable: The convert_urns_to_lowercase setting should be treated as fixed for the life of an ingestion pipeline. Changing it after assets have been ingested requires a manual cleanup step as described above.
Soft deletes are reversible: Soft-deleted assets can be restored. If the cleanup step causes unintended deletions, contact DataHub Support for assistance with restoration.
Stateful ingestion limitation: Stateful ingestion's orphan-removal logic uses exact URN string matching. It cannot detect that an uppercase URN and a lowercase URN refer to the same logical asset.
The ignore_old_state: true flag must be removed after the cleanup run. Leaving it enabled permanently defeats the purpose of stateful ingestion, as it will rebuild the checkpoint from scratch on every run.