Enabling Lineage Between dbt and AWS Glue in DataHub: URN Alignment and Column Lineage – DataHub Customer Support Portal

```html

Area: Ingestion Issues

Sub-Area: Cross-Platform Lineage Configuration (dbt + AWS Glue)

Issue

When ingesting both AWS Glue and dbt into DataHub, lineage between the two platforms may fail to appear or be incomplete. The most common causes are: (1) URN mismatches between the Glue and dbt ingestion sources, where the number of path components in the dataset identifier differs (3-part vs. 4-part URNs); (2) Glue's emit_s3_lineage flag overwriting lineage edges created by dbt ingestion; and (3) missing or misconfigured target_platform settings that prevent DataHub from recognizing that a dbt model materializes to a Glue table. Additionally, column-level lineage between dbt and Glue-backed models may be absent if compiled SQL is not available in the dbt manifest or if the SQL dialect is not correctly resolved.

Error Messages

sql_parser_successes: 0
sql_parser_skipped_missing_code: 131
fineGrainedLineages (Model): 0
The run results file is from a `dbt docs generate` command, instead of a build/run/test command. Skipping this file.
ValueError: Unknown dialect 'glue'

You Might Be Asking

Why is there no lineage showing between my Glue datasets and my dbt models in DataHub?
Why does the Glue URN have 3 path parts but the dbt URN has 4 path parts for the same table?
How do I configure target_platform and target_platform_instance in a dbt recipe to point to Glue?
Why does running Glue ingestion clear the lineage edges created by dbt ingestion?
Why is column-level lineage (CLL) not being generated for my dbt models that materialize to Glue?
Will glossary term propagation work across dbt and Glue sibling datasets once column lineage is established?

Solution

Work through the following steps in order. Each step addresses a distinct configuration layer required for end-to-end Glue ↔ dbt lineage in DataHub.

Set target_platform: glue in the dbt recipe
Tell the dbt connector which platform the models materialize to. Without this, DataHub cannot stitch lineage between dbt nodes and Glue tables.
```
source:
  type: dbt
  config:
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    target_platform: glue
    target_platform_instance: <your-glue-platform-instance>
    # Example: target_platform_instance: prod-usw2-glue
```
The value of target_platform_instance must exactly match the platform instance configured in your Glue ingestion recipe. If your Glue recipe does not set a platform_instance, omit this field from the dbt recipe as well.
Resolve URN mismatches using include_database_name
A common symptom after step 1 is that Glue ingestion generates 3-part URNs while dbt ingestion generates 4-part URNs for the same underlying table. For example:
- Glue URN (3-part): urn:li:dataset:(urn:li:dataPlatform:glue,<platform-instance>.<schema>.<table>,PROD)
- dbt URN (4-part): urn:li:dataset:(urn:li:dataPlatform:glue,<platform-instance>.<database>.<schema>.<table>,PROD)
DataHub treats these as two different datasets. To drop the database component from the dbt-generated URN and match the Glue 3-part format, add include_database_name: false to your dbt recipe:
```
source:
  type: dbt
  config:
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    target_platform: glue
    target_platform_instance: <your-glue-platform-instance>
    include_database_name: false
```
After this change, re-run dbt ingestion and verify that the URNs produced by both connectors now match in DataHub.
Disable emit_s3_lineage in the Glue recipe to prevent lineage overwriting
By default, the Glue connector may emit S3-based lineage edges. When Glue ingestion runs, these S3 lineage edges can overwrite or clear the lineage edges that dbt ingestion previously established. To prevent this, set emit_s3_lineage: false in your Glue recipe:
```
source:
  type: glue
  config:
    aws_region: <your-aws-region>
    platform_instance: <your-glue-platform-instance>
    emit_s3_lineage: false
    # Add extract_transforms: true if you want ETL job-level lineage
    extract_transforms: true
```
Note: Only disable emit_s3_lineage if you are relying on dbt ingestion to supply lineage edges for Glue-backed tables. If S3 lineage is needed for other datasets, evaluate the impact before disabling globally.
Enable column-level lineage in the dbt recipe
To propagate glossary terms and other metadata downstream via column-level lineage, enable column lineage in the dbt recipe:
```
source:
  type: dbt
  config:
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    target_platform: glue
    target_platform_instance: <your-glue-platform-instance>
    include_database_name: false
    include_column_lineage: true
```
After enabling this flag and re-running ingestion, verify column-level lineage edges appear in the DataHub UI for the relevant datasets.
Ensure compiled SQL is present in the dbt manifest
Column-level lineage for dbt models is derived by parsing the compiled SQL in the dbt manifest. If compiled_code is absent (for example, when the manifest was generated via dbt docs generate rather than dbt build or dbt run), the SQL parser is skipped entirely and no column lineage is produced. You will see the following in your ingestion run report:
```
sql_parser_skipped_missing_code: <N>   # N models had no compiled_code
fineGrainedLineages (Model): 0          # zero models have column lineage
Warning: The run results file is from a `dbt docs generate` command,
instead of a build/run/test command. Skipping this file.
```
To resolve this, generate the manifest and catalog by running dbt build or dbt run (not dbt docs generate) before ingestion:
```
# Correct approach — produces compiled_code in manifest.json
dbt run --profiles-dir /path/to/profiles
dbt docs generate --profiles-dir /path/to/profiles
```
Then point your DataHub dbt recipe at the manifest and catalog produced by this run.

Complete example recipes

The following shows a minimal working pair of recipes for Glue and dbt ingestion with lineage alignment:

Glue recipe (glue_recipe.yml):

source:
  type: glue
  config:
    aws_region: <your-aws-region>
    platform_instance: <your-glue-platform-instance>
    emit_s3_lineage: false
    extract_transforms: true

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.acryl.io/api/gms"
    token: "<your-datahub-token>"

dbt recipe (dbt_recipe.yml):

source:
  type: dbt
  config:
    manifest_path: /path/to/manifest.json
    catalog_path: /path/to/catalog.json
    target_platform: glue
    target_platform_instance: <your-glue-platform-instance>
    include_database_name: false
    include_column_lineage: true

sink:
  type: datahub-rest
  config:
    server: "https://<your-instance>.acryl.io/api/gms"
    token: "<your-datahub-token>"

Validate lineage and glossary term propagation
1. Re-run Glue ingestion first, then dbt ingestion.
2. In the DataHub UI, open a dbt-backed dataset and confirm the Lineage tab shows upstream and downstream edges to the corresponding Glue datasets.
3. Confirm column-level lineage is visible under the Column Lineage view.
4. To test glossary term propagation: manually apply a glossary term to a column on an upstream dataset, then navigate to your Automations page, find the Propagate Glossary Terms automation, and click Initialize to trigger a backfill. Verify the term appears on the same column name in downstream datasets.

Additional Notes

Glue job type prerequisite: The Glue connector can only extract ETL job-level lineage from jobs created using the "Generate classic script" option in AWS Glue. Jobs created as fully custom scripts do not expose the metadata needed for lineage extraction, regardless of DataHub configuration.

Known limitation — SQL query parameter: If a Glue job uses the SQL query parameter (rather than dbtable) for reads, target_platform_configs will not apply to those upstream references. Only the direct table-name path supports platform configuration mapping currently.

Unknown dialect error for Glue: In some DataHub versions, the dbt connector passes target_platform as the SQL dialect string to the SQL parser. Because glue is not a recognized dialect in sqlglot, a ValueError: Unknown dialect 'glue' is raised and column lineage parsing fails even when compiled SQL is present. This is a known bug tracked in the DataHub open-source repository (see GitHub PR #17728). If you encounter this error after ensuring compiled SQL is available, check whether a patched version of DataHub is available for your deployment.

Ingestion ordering: Run Glue ingestion before dbt ingestion. The dbt connector relies on the Glue datasets already existing in DataHub to correctly create sibling relationships and lineage edges.

Sibling traversal in the UI: Because lineage edges are attached to dbt nodes while the UI defaults to displaying the Glue sibling (merged view), full end-to-end lineage traversal across multiple layers may require switching to the dbt sibling view at each hop. This is a known UI behavior for sibling dataset pairs.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation