MS SQL Server Ingestion: Impact of convert_column_urns_to_lowercase on Lineage and Display – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: MS SQL Server Connector — URN Casing and Column Lineage

Issue

When enabling the convert_column_urns_to_lowercase setting in an MS SQL Server ingestion recipe, users may encounter questions about how previously ingested lineage is affected, whether a backfill job runs automatically, how cross-source lineage behaves when only some sources have the setting enabled, and whether it is possible to display original column casing in the UI while still using lowercase backend URN identifiers.

You Might Be Asking

What happens to existing column-level lineage when I enable convert_column_urns_to_lowercase for the first time?
Will lineage break if only one source in a cross-platform pipeline has this setting enabled?
Why are there no standalone schemaField entities in my metadata output file after ingestion?
Can I lowercase the backend URN identifiers while still showing the original column casing in the DataHub UI?

Solution

Understand what convert_column_urns_to_lowercase does.
When this flag is set to true, the ingestion pipeline rewrites column URNs to use lowercase field names. For example:
```
# Before (mixed-case URN)
urn:li:schemaField:(urn:li:dataset:(...),UserID)

# After (lowercased URN)
urn:li:schemaField:(urn:li:dataset:(...),userid)
```
This rewrite applies to the schemaMetadata aspect on dataset entities and to any column-level lineage references generated by that connector.
Know that there is no automatic backfill job.
Enabling this setting does not trigger any backfill or migration of previously ingested data. The new lowercase URNs are only created when the next ingestion run executes. Old uppercase URNs and their associated lineage edges remain in the catalog until they are overwritten or soft-deleted by subsequent ingestion runs.
Understand same-source vs. cross-source lineage behavior.
- Same-source lineage (e.g., SQL Server to SQL Server): Lineage will typically continue to resolve correctly. The ingestion run generates new lowercase URNs for both upstream and downstream column references, so stitching remains consistent.
- Cross-source lineage (e.g., SQL Server → Snowflake, SQL Server → BigQuery): Lineage is likely to break if only one source has convert_column_urns_to_lowercase: true. One platform will emit uppercase URNs while the other emits lowercase URNs, causing the lineage join to fail because the URNs no longer match.
Best practice: If you enable convert_column_urns_to_lowercase: true in any recipe that participates in cross-platform column lineage, enable it consistently in all related recipes.
```
# Example: SQL Server recipe snippet
source:
  type: mssql
  config:
    # ... connection config ...
    convert_column_urns_to_lowercase: true

# Example: Snowflake recipe snippet (must match)
source:
  type: snowflake
  config:
    # ... connection config ...
    convert_column_urns_to_lowercase: true
```
Understand why standalone schemaField entities do not appear in metadata output files.
This is expected behavior. Schema field metadata is stored as part of the schemaMetadata aspect on the dataset entity, not as independently emitted schemaField MCPs. When you inspect a metadata file produced by the ingestion, you will see dataset entities with a schemaMetadata aspect containing all field definitions (with correctly lowercased names) rather than separate top-level schemaField entries.

The term "first-class schemaField entities" in the DataHub documentation refers to how DataHub uses schemaField URNs internally for fine-grained column-level lineage resolution — not to standalone entities that connectors emit independently. Your metadata file output is correct.
Understand the current limitation around displaying original column casing.
At this time, it is not possible to display the original mixed-case column name in the DataHub UI while simultaneously using a lowercased backend URN for that column. Lowercasing the URN identifier also lowercases how the column name appears in the catalog. This behavior differs from dataset-level URN lowercasing, which provides more flexibility.

If this capability is important to your use case, consider submitting a feature request to the DataHub team. As a workaround, evaluate whether column URN lowercasing is strictly necessary for your environment, or limit its use to sources where collision prevention is a confirmed requirement.

Additional Notes

Enabling convert_column_urns_to_lowercase is a one-way transformation that affects URN identity. Once lineage and metadata have been ingested with lowercase URNs, reverting to mixed-case URNs will produce new, distinct URNs and may result in duplicate or orphaned lineage edges until the old data is cleaned up. Plan carefully before enabling this setting in a production environment. The behavior described here applies to DataHub Cloud and self-hosted deployments using the current DataHub ingestion CLI. Always test recipe changes against a staging or metadata-file dry-run before applying them to production.

Issue

You Might Be Asking

Solution

Additional Notes

Related Documentation