Column-Level Lineage Missing for dbt Models with AWS Glue as Target Platform – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt Ingestion / Column-Level Lineage / SQL Parsing

Issue

When ingesting a dbt project into DataHub with AWS Glue configured as the target_platform, table-level lineage is generated successfully but column-level lineage fails for every dbt model. The ingestion report shows 100% SQL parser failures and zero successes. The root cause is that the SQL parser was using the storage platform name ("glue") as the SQL dialect identifier when invoking the underlying SQL parser library, which does not recognize "glue" as a valid dialect. AWS Glue executes Trino/Presto-compatible SQL, so the parser must be directed to use the "trino" dialect instead.

Error Messages

Error parsing SQL to generate column lineage model.<project>.<model_name> <class 'ValueError'>: Unknown dialect 'glue'
sql_parser_parse_failures: <N>
sql_parser_successes: 0

You Might Be Asking

Why does table-level lineage work but column-level lineage fails completely when using dbt with AWS Glue?
What does "Unknown dialect 'glue'" mean in my dbt ingestion logs?
Is there a workaround for missing column-level lineage when using dbt with AWS Glue as the target platform?
Which DataHub CLI version includes the fix for the glue SQL dialect error?
Should I set sql_parser_dialect in my recipe to work around this issue?

Root Cause

The dbt ingestion code was resolving the SQL dialect for column-level lineage parsing from the storage platform name (e.g., "glue") rather than from the adapter_type field in the dbt manifest (metadata.adapter_type). Because "glue" is not a recognized SQL dialect in the underlying SQL parser library, every model's SQL failed to parse, resulting in a 100% failure rate for column-level lineage. Table-level lineage does not require SQL parsing and was therefore unaffected.

Two separate fixes addressed this:

PR #17728 — Changes the dialect resolution logic in dbt_common.py to use node.dbt_adapter (from the dbt manifest's metadata.adapter_type) instead of the storage platform name.
PR #17893 — Adds additional Glue-related lineage and view classification improvements to complement the dialect fix.

Both fixes are included in acryl-datahub version 1.7.0 and later.

Solution

Upgrade the DataHub CLI to version 1.7.0 or later.
Both PR #17728 and PR #17893 are included in acryl-datahub 1.7.0. Upgrade your CLI:
```
pip install "acryl-datahub>=1.7.0"
```
For DataHub Cloud deployments, update the pinned CLI version in the advanced settings of your dbt ingestion source to 1.7.0 (or remove any custom pip override to revert to the default managed version).

Do NOT manually set sql_parser_dialect in your recipe.

This field is not a valid configuration option in the dbt ingestion source and will cause a config validation error, preventing ingestion from running entirely. Remove it if present:

# INCORRECT — remove this from your recipe
source:
  type: dbt
  config:
    target_platform: glue
    # sql_parser_dialect: trino   ← DO NOT add this; it is not a valid field

The correct recipe requires no special dialect override once you are on version 1.7.0+:

source:
  type: dbt
  config:
    target_platform: glue
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    # No sql_parser_dialect override needed

Re-run your dbt ingestion.
After upgrading, trigger a new ingestion run. Verify success by checking the ingestion report for the following signals:
- sql_parser_successes is greater than zero.
- No Unknown dialect 'glue' errors appear in the logs.
- Column-level lineage is visible in the DataHub UI for affected dbt models.
It is recommended to validate on a staging environment before re-running in production.
Confirm your dbt manifest contains the adapter type.
The fix in PR #17728 reads the SQL dialect from the metadata.adapter_type field in your dbt manifest. Verify this field is present:
```
# Example manifest.json (truncated)
{
  "metadata": {
    "dbt_schema_version": "...",
    "adapter_type": "trino",   // ← must be present and correct
    ...
  },
  "nodes": { ... }
}
```
If adapter_type is missing from your manifest, regenerate it by running dbt compile or dbt docs generate with a supported dbt-core version.

Additional Notes

This issue affects all DataHub CLI versions prior to 1.7.0 when using dbt with target_platform: glue. There is no supported workaround on older versions; upgrading is the only resolution.
This is a pre-existing feature gap, not a regression. The "glue" platform name was never mapped to a recognized SQL dialect, so column-level lineage for dbt+Glue has never worked in versions before the fix.
Other target platforms such as Snowflake, Redshift, and BigQuery are not affected because their platform names happen to match recognized SQL dialect identifiers in the parser library.
Ephemeral dbt models do not have physical tables in the target platform. Per DataHub documentation, queries on ephemeral models are skipped because there is no dataset to link to. If you observe lineage differences for ephemeral models between environments (e.g., staging vs. production), also check whether node_name_pattern filters or different dbt artifact sources are causing the discrepancy before attributing it to the ephemeral model limitation.
For DataHub Cloud customers, CLI upgrades are managed by the DataHub Cloud team. Contact your support representative if you need to expedite the upgrade or validate a specific version.