URN Construction Bug in S3 Lineage Helper Causes Crashes When Paths Contain Reserved Characters – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: S3 URN Encoding / Lineage Helper Functions

Issue

The make_s3_urn and make_s3_urn_for_lineage helper functions in datahub/ingestion/source/aws/s3_util.py construct dataset URNs by directly interpolating the raw S3 path into an f-string without applying any URL encoding. S3 object keys that contain characters reserved in the URN specification — such as parentheses ( and ) — are embedded verbatim, producing an invalid URN that the SDK's own parser immediately rejects. This causes a hard crash during ingestion for any connector that calls these helpers, including Snowflake lineage, Glue, Athena, SageMaker, and Databricks Unity Catalog. The bug is confirmed present through at least SDK version 1.7.0.10 and requires a code-level fix; no configuration workaround or version upgrade resolves it on its own.

Error Messages

InvalidUrnError: DatasetUrn name contains reserved characters
InvalidUrnError: Expecting a Urn but got urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/data/folder(1)/file.parquet,PROD)

You Might Be Asking

Why does Snowflake (or Glue / Athena / SageMaker / Unity Catalog) lineage ingestion crash only for certain S3 paths?
Why does upgrading the SDK not fix the InvalidUrnError on S3 URN construction?
How do I generate a valid DataHub dataset URN for an S3 path that contains parentheses or other special characters?
Which connectors are affected by the make_s3_urn encoding bug?

Root Cause

The defective code in s3_util.py builds the URN with a plain f-string:

def make_s3_urn(s3_uri: str, env: str, remove_extension: bool = True) -> str:
    s3_name = strip_s3_prefix(s3_uri)
    if s3_name.endswith("/"):
        s3_name = s3_name[:-1]
    name, extension = os.path.splitext(s3_name)
    if remove_extension and extension != "":
        extension = extension[1:]  # remove the dot
        return f"urn:li:dataset:(urn:li:dataPlatform:s3,{name}_{extension},{env})"
    return f"urn:li:dataset:(urn:li:dataPlatform:s3,{s3_name},{env})"

No encoding step is applied. The SDK's own UrnEncoder defines the following as reserved characters that must be percent-encoded inside a URN segment:

# datahub/utilities/urn_encoder.py
RESERVED_CHARS = {",", "(", ")", "␟"}

The DatasetUrn class constructor applies UrnEncoder.encode_string() automatically, but make_s3_urn bypasses the class entirely and constructs the string directly, so parentheses and other reserved characters land in the URN unescaped. A subsequent call to DatasetUrn.from_string() — or any internal parse — then raises InvalidUrnError.

The following modules in the SDK all import and call this helper, and are therefore all affected by the same bug:

ingestion/source/aws/glue.py
ingestion/source/aws/sagemaker_processors/jobs.py
ingestion/source/snowflake/snowflake_lineage_v2.py
ingestion/source/snowflake/snowflake_schema_gen.py
ingestion/source/snowflake/snowflake_stages.py
ingestion/source/sql/athena.py
ingestion/source/unity/source.py

Solution

Confirm the bug (optional reproduction). Run the following three lines against your installed SDK version. If you receive InvalidUrnError, you are hitting this bug:

from datahub.ingestion.source.aws.s3_util import make_s3_urn_for_lineage
from datahub.utilities.urns.dataset_urn import DatasetUrn

DatasetUrn.from_string(
    make_s3_urn_for_lineage("s3://my-bucket/data/folder(1)/file.parquet", "PROD")
)
# Raises: InvalidUrnError: DatasetUrn name contains reserved characters

Apply the immediate workaround — construct the URN through DatasetUrn directly (recommended). The DatasetUrn constructor already calls UrnEncoder.encode_string() internally, so it produces a correctly encoded URN without any manual escaping:

from datahub.ingestion.source.aws.s3_util import strip_s3_prefix
from datahub.utilities.urns.dataset_urn import DatasetUrn

s3_uri = "s3://my-bucket/data/folder(1)/file.parquet"
env = "PROD"

urn = DatasetUrn(
    platform="s3",
    name=strip_s3_prefix(s3_uri),
    env=env,
).urn()

# Produces: urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/data/folder%281%29/file.parquet,PROD)
# This URN round-trips through DatasetUrn.from_string() without error.
print(urn)

Alternative workaround — encode the name manually before interpolation. If you must construct the URN string yourself, apply UrnEncoder.encode_string() to the S3 path before embedding it:

from datahub.ingestion.source.aws.s3_util import strip_s3_prefix
from datahub.utilities.urn_encoder import UrnEncoder

s3_uri = "s3://my-bucket/data/folder(1)/file.parquet"
env = "PROD"

encoded_name = UrnEncoder.encode_string(strip_s3_prefix(s3_uri))
urn = f"urn:li:dataset:(urn:li:dataPlatform:s3,{encoded_name},{env})"

# Produces the identical result to the DatasetUrn approach above.
print(urn)

Verify the output URN parses correctly. Both workarounds above produce the same encoded URN. Confirm it is valid by parsing it back:

from datahub.utilities.urns.dataset_urn import DatasetUrn

# Should not raise — prints the URN if valid.
parsed = DatasetUrn.from_string(urn)
print(parsed)

Apply the upstream SDK fix when it ships. A patch to make_s3_urn that applies UrnEncoder.encode_string() inside the function before string interpolation has been raised with the engineering team (see GitHub PR #20220). Once merged and released, the helper will behave correctly for all callers and no application-level workaround will be needed. The fix encodes ( → %28, ) → %29, and other reserved characters, while leaving forward slashes intact. Paths that do not contain reserved characters are unaffected.

When upgrading to a version that includes the fix, URNs created via the workarounds above will continue to match — both approaches produce the same percent-encoded form that the fixed helper will also produce — so no entity re-keying or metadata cleanup is required for paths that previously had no valid URN.

Additional Notes

Affected SDK versions: Confirmed in acryl-datahub 1.6.0.17, 1.7.0.8, and 1.7.0.10. The function is unchanged across these versions; upgrading alone does not resolve the issue.
Double-encoding risk: If you previously applied a manual workaround that pre-escaped parentheses before passing the path to make_s3_urn_for_lineage, upgrading to a version that includes the upstream fix will result in double-encoding (e.g., %2528 instead of %28). Review any pre-processing logic before upgrading.
Incremental lineage consideration: If any ingestion run succeeded in writing a broken (unencoded) URN to DataHub before this issue was identified — which is more likely with connectors such as Glue that do not raise on write — an old edge or dataset entity with the unencoded URN may remain alongside the new correctly-encoded one when incremental_lineage: true is enabled. In that case, the stale entity should be removed manually.
Spark lineage: The Spark lineage connector constructs S3 dataset URNs independently and does not call make_s3_urn. It may be affected by a similar encoding gap; this is under separate investigation.
URN reserved character reference: The full list of characters that must be encoded inside a URN segment is documented in the DataHub URN specification: commas ,, parentheses ( and ), and the unit separator ␟.