dbt tags_to_structured_properties Transformer Not Working Due to Stateful Ingestion Skipping – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: dbt Transformers / Stateful Ingestion

Issue

When using the tags_to_structured_properties transformer with the dbt ingestion source, structured properties may not be populated on existing datasets even though tags are visible in the DataHub UI. The ingestion run completes quickly and reports no errors, but the structured properties fields remain empty. This occurs because stateful ingestion detects that the entities have not changed since the last run and skips re-emitting the globalTags metadata change proposals (MCPs). Since the transformer only processes MCPs as they flow through the active pipeline run, it never has an opportunity to act on the pre-existing tags. Entities that were first ingested without the transformer configured are particularly susceptible to this silent skip behavior.

You Might Be Asking

Why are my dbt tags visible in DataHub but no structured properties are being created?
Why did the tags_to_structured_properties transformer work for newly ingested entities but not for existing ones?
Why does my dbt ingestion run finish in seconds when the transformer is enabled?
How do I force the transformer to process tags on entities that were already in DataHub before the transformer was added to my recipe?
Does tags_to_structured_properties support column-level dbt tags from column_meta_mapping?

Solution

The fix requires a one-time forced full ingestion run with stateful ingestion disabled. This forces DataHub to re-emit all globalTags MCPs, giving the transformer the opportunity to process them and write structuredProperties aspects. Follow these steps:

Open your dbt ingestion recipe and temporarily disable stateful ingestion:

source:
  type: dbt-cloud  # or dbt
  config:
    # ... your existing config ...
    stateful_ingestion:
      enabled: false  # temporarily disabled for this run

Ensure your transformers block is present and correctly configured. If running from the DataHub UI, use the raw YAML editor — the UI ingestion form may silently drop the transformers block. A complete example:

source:
  type: dbt-cloud
  config:
    account_id: ''
    project_id: ''
    job_id: ''
    target_platform: bigquery
    token: '${dbt_token}'
    env: PROD
    convert_urns_to_lowercase: true
    platform_instance: ''
    tag_prefix: ""
    meta_mapping:
      data.classification:
        match: ".*"
        operation: add_tag
        config:
          tag: "data.classification:{{ $match }}"
      tier:
        match: ".*"
        operation: add_tag
        config:
          tag: "tier:{{ $match }}"
    stateful_ingestion:
      enabled: false   # disabled for the bootstrap run only

transformers:
  - type: "tags_to_structured_properties"
    config:
      process_key_value_tags: true
      key_value_separator: ":"
      key_value_property_prefix: ""
      remove_original_tags: false
      semantics: PATCH

Run the ingestion. To capture detailed output for troubleshooting, use the --debug flag:
```
datahub ingest -c your-dbt-recipe.yaml --debug
```
In the debug output, confirm that the transformer logs processing tags and emitting structuredProperties aspects for your datasets.
After the successful bootstrap run, re-enable stateful ingestion in your recipe:
```
stateful_ingestion:
  enabled: true
  remove_stale_metadata: true
```
Going forward, structured properties will persist in DataHub. When a meta field value changes in dbt, the next run will re-emit the globalTags MCP and the transformer will update the corresponding structured property automatically.

Known Limitation: Column-Level Tags from `column_meta_mapping`

The tags_to_structured_properties transformer processes standalone globalTags aspects emitted on schemaField entity URNs (urn:li:schemaField:...). However, the dbt source embeds column-level tags inside the schemaMetadata aspect on the dataset URN as nested SchemaField.globalTags — it does not emit standalone schemaField MCPs. Because of this architectural mismatch, the transformer cannot see or process column-level tags from column_meta_mapping. Only dataset-level tags configured via meta_mapping are supported by this transformer.

If you require column-level structured properties from dbt, consider using the native add_to_structured_properties operation available in newer DataHub CLI versions, which provides a direct path from dbt model metadata to structured properties without relying on an intermediate tag conversion step.

Multi-Value Field Formatting in dbt Schema

If a structured property field (such as appid) accepts multiple values, define it as a YAML list in your dbt schema.yml rather than a comma-separated string. A comma-separated string will be ingested as a single concatenated value, not as two distinct entries:

# Correct: YAML list for multi-value structured properties
models:
  - name: your_model_name
    config:
      meta:
        appid:
          - APP00001
          - APP00002

# Incorrect: comma-separated string (ingested as one value)
models:
  - name: your_model_name
    config:
      meta:
        appid: "APP00001,APP00002"

Additional Notes

This behavior affects any ingestion source — not just dbt — where entities were first ingested without the tags_to_structured_properties transformer. The one-time forced full run workaround applies broadly.
Ingestion runs launched from a local CLI (datahub ingest -c file.yaml) do not register in the managed ingestion history visible in the DataHub UI. Use managed ingestion via the raw YAML editor if you need run logs to be accessible from the DataHub platform.
Column-level tag-to-structured-property conversion via column_meta_mapping is a known product gap. Monitor the DataHub release notes for updates on native structured property support at the column level.
The native add_to_structured_properties operation in the DataHub CLI is the recommended long-term approach for dbt-to-structured-property mapping and avoids the stateful ingestion bootstrapping problem described here.