tags_to_ignore_sampling Configuration: Using Tag Name vs. Display Name for Profiling Exclusion – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Profiling Configuration

Issue

When configuring tags_to_ignore_sampling in a DataHub profiling recipe, columns tagged with privacy or governance tags continue to be sampled even though the configuration is accepted without validation errors. This occurs because the configuration field requires the tag's name (the identifier portion of the tag URN) rather than the tag's display name as shown in the DataHub UI. Using the human-readable display name results in no match at runtime, so the sampling exclusion is silently skipped. Additionally, a secondary bug exists where column-level tags applied via ingestion or classification (stored in SchemaMetadata) may not be evaluated at all — only tags applied manually through the UI (stored in EditableSchemaMetadata) are currently checked by the profiler.

Error Messages

No validation error is raised — the misconfiguration fails silently at runtime.

You Might Be Asking

Why is my column still being profiled/sampled even though it has a privacy tag and I've listed that tag in tags_to_ignore_sampling?
Should I use the tag display name or the tag URN in tags_to_ignore_sampling?
Does tags_to_ignore_sampling work with tags applied via ingestion or classification, or only tags added manually in the UI?
Why does my tags_to_ignore_sampling configuration accept any value without error even when it doesn't work?

Solution

Identify the correct tag name to use in your configuration.
The tags_to_ignore_sampling field matches against the tag name — the portion of the tag URN that follows urn:li:tag: — not the display name shown in the DataHub UI.
- Tag URN example: urn:li:tag:your_org.tag.gov_privacy_category:identifier
- Tag display name (UI): GOV_PRIVACY_CATEGORY: IDENTIFIER ← do not use this
- Correct value for config: your_org.tag.gov_privacy_category:identifier ← use this
To find the tag name for any tag, navigate to the tag in the DataHub UI and inspect its URN. Everything after urn:li:tag: is the value you should use.

Update your profiling recipe to use the tag name.

Replace any display names in tags_to_ignore_sampling with the corresponding tag name extracted from the URN:

profiling:
  enabled: true
  profile_table_level_only: false
  use_sampling: true
  sample_size: 10000
  tags_to_ignore_sampling:
    - your_org.tag.gov_privacy_category:identifier
    - your_org.tag.gov_privacy_category:sensitive
  field_sample_values_limit: 20   # Keep this low; default is 20, max recommended is 50-100
  profile_if_updated_since_days: 7
  operation_config:
    lower_freq_profile_enabled: true
    profile_day_of_week: 0

Replace your_org.tag.gov_privacy_category:identifier and your_org.tag.gov_privacy_category:sensitive with the actual tag name portions from your own tag URNs.

Verify how the tag was applied to the column.
Even with the correct tag name, a secondary limitation exists in the profiler code:
- Tags applied manually through the DataHub UI are stored in EditableSchemaMetadata and will be evaluated by tags_to_ignore_sampling.
- Tags applied via ingestion pipelines or DataHub's classification feature are stored in SchemaMetadata and are currently not evaluated — this is a known product bug affecting both the SQLAlchemy profiler (sqlalchemy_profiler.py) and the Great Expectations profiler (ge_data_profiler.py).
To determine which applies to your column, check the tag indicator in the DataHub UI column view. If the tag shows an ingestion-source icon (rather than a "manually added" indicator), the tag is stored in SchemaMetadata and the exclusion will not take effect until the bug is resolved.

Workaround for ingestion-applied tags: Manually re-apply the tag through the DataHub UI so it is stored in EditableSchemaMetadata, which the profiler does evaluate.

Apply additional profiling performance recommendations.

While troubleshooting, consider these settings to reduce load and improve reliability:

profiling:
  max_workers: 2          # Reduces parallel DB connections; default is 4
  field_sample_values_limit: 20   # Keep at default (20) or no more than 50-100

Re-run ingestion and verify results.
After updating the configuration, trigger a new ingestion run. Navigate to the affected table's Columns view in the DataHub UI and confirm that sample values are no longer displayed for columns carrying the excluded tags.

Additional Notes

The bug affecting column-level tag lookups from SchemaMetadata (as opposed to EditableSchemaMetadata) was identified and a fix was submitted to the DataHub open-source repository. The fix is expected to be included in CLI version v1.6.0.11 or later. Until the patched CLI version is available in your environment, the manual UI tag workaround described above is the recommended mitigation. This behavior affects both DataHub OSS and DataHub Cloud deployments running versions prior to the fix. The tags_to_ignore_sampling field performs no validation at parse time, so incorrect values (including non-existent tag names or display names) are silently ignored — always verify exclusions are working by inspecting the profiled column output after ingestion.

Issue

Error Messages

You Might Be Asking

Solution

Additional Notes

Related Documentation