Managing Search Results and Filtering by Environment – DataHub Customer Support Portal

Area: Product Issues

Sub-Area: Search and Filtering

Issue

Users experience problems with search result ranking and environment filtering in DataHub. Search results may not properly prioritize entities by usage metrics, and environment filters may not display all available environments when searching across different platforms like Airflow pipelines.

Error Messages

Usage information is not taken into account in search ranking
Environment filter only shows limited options despite having multiple environments ingested

You Might Be Asking

Why aren't my high-usage datasets appearing first in search results?
Why can't I see all my environments in the search filter dropdown?
How do I enable sort controls in search results?
Why do my pipeline environments not appear in the environment filter?

Solution

For Search Ranking Issues:

Verify that usage statistics are being properly ingested by checking if your datasets have usage data populated in the system
For DataHub Cloud instances experiencing ranking problems, contact DataHub Support as search ranking configurations are managed by Acryl and may require backend fixes
Request enablement of sort controls by asking support to set the feature flag showDatasetFeaturesSearchSortOptions: true
If you're on an older version experiencing ranking regression, upgrade to DataHub v0.3.17.1 or later which includes improved search ranking algorithms

For Environment Filtering Issues:

Ensure your ingestion configuration uses valid FabricType values for environments. Valid values include:
```
DEV, TEST, QA, UAT, EI, PRE, STG, NON_PROD, PROD, CORP, RVW, PRD, TST, SIT, SBX, SANDBOX
```

For Airflow pipelines, update your cluster configuration to use a valid FabricType:

# Instead of 'staging', use:
AIRFLOW__DATAHUB__CLUSTER=stg

# Instead of 'production', use:  
AIRFLOW__DATAHUB__CLUSTER=prod

If changing cluster values, clean up existing entities first to avoid duplicates:

# Preview entities that will be deleted
datahub delete --platform airflow --env  --dry-run

# Execute the deletion
datahub delete --platform airflow --env

Re-trigger ingestion after updating configuration to populate entities with correct environment values

Verification Steps:

Check that environment filters now show all expected environments in the search interface
Verify search results prioritize high-usage entities appropriately
Confirm sort controls are available if requested

Additional Notes

Environment filtering for pipeline entities (like Airflow DataFlows) may behave differently than for datasets due to different facet field configurations. The cluster value in URNs serves as identification while the env field in the dataFlowInfo aspect drives search filtering. Search ranking in DataHub Cloud combines textual relevance with usage metrics, but exact weighting is managed by the platform team. Changes to cluster values will modify URNs, so proper cleanup of existing entities is important to prevent duplicates.