Dataset Profile and Statistics Missing

Dataset profiles (row counts, column statistics, data distributions) are not appearing in DataHub after ingestion. This typically occurs when profiling is disabled, sampling is configured incorrectly, or profiling queries fail.

You Might Be Asking:

• How do I enable dataset profiling?
• Why are my row counts showing as zero?
• Can I profile without scanning all data?

1. Enable basic profiling in ingestion:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    profiling:
      enabled: true

      # Table-level profiling (fast)
      profile_table_level_only: false

      # Row sampling
      profile_table_row_limit: 50000
      max_number_of_fields_to_profile: 20

2. Configure column-level statistics:

source:
  config:
    profiling:
      enabled: true

      # What to profile
      include_field_null_count: true
      include_field_min_value: true
      include_field_max_value: true
      include_field_mean_value: true
      include_field_median_value: true
      include_field_stddev_value: true
      include_field_quantiles: true
      include_field_distinct_count: true
      include_field_histogram: true
      include_field_sample_values: true

      # Sampling
      max_number_of_fields_to_profile: 50

3. Profile specific tables only:

source:
  config:
    profiling:
      enabled: true

      # Filter which tables to profile
      allow_deny_patterns:
        allow: ["analytics\..*", "reporting\..*"]
        deny: [".*_temp$", ".*_staging$"]

      # Size limits
      profile_table_size_limit: 5000000  # Only tables < 5M rows

4. Emit custom profiling data:

Example for Snowflake. This approach can be adapted for any data source/connector.

from datahub.emitter.mce_builder import make_dataset_urn
from datahub.metadata.schema_classes import (
    DatasetProfileClass,
    DatasetFieldProfileClass
)
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter("http://localhost:8080")

# Create dataset profile
dataset_profile = DatasetProfileClass(
    timestampMillis=int(time.time() * 1000),
    rowCount=1000000,
    columnCount=25,

    # Field profiles
    fieldProfiles=[
        DatasetFieldProfileClass(
            fieldPath="user_id",
            uniqueCount=50000,
            uniqueProportion=0.05,
            nullCount=0,
            nullProportion=0.0,
            min="1",
            max="1000000",
            mean="500000",
            median="500000",
            stdev="288675",
            sampleValues=["123", "456", "789"]
        ),
        DatasetFieldProfileClass(
            fieldPath="email",
            uniqueCount=50000,
            nullCount=100,
            nullProportion=0.001,
            sampleValues=["user@example.com", "test@test.com"]
        )
    ]
)

emitter.emit_mcp(
    entity_urn=make_dataset_urn("snowflake", "db.schema.table"),
    aspect_name="datasetProfile",
    aspect=dataset_profile
)

5. Query profile data:

query getDatasetProfile {
  dataset(urn: "YOUR_URN") {
    datasetProfiles(limit: 1) {
      rowCount
      columnCount
      timestampMillis

      fieldProfiles {
        fieldPath
        uniqueCount
        uniqueProportion
        nullCount
        nullProportion
        min
        max
        mean
        median
        stdev
        sampleValues
      }
    }
  }
}

6. Troubleshoot profiling failures:

Example for Snowflake. The troubleshooting approach can be adapted for other data sources/connectors.

# Check ingestion logs for profiling errors
datahub ingest -c recipe.yml 2>&1 | grep -i "profil"

# Common issues:
# 1. Insufficient permissions to run queries
# 2. Query timeout on large tables
# 3. Memory issues with histogram generation

# Test profiling query manually
# Snowflake example:
SELECT 
  COUNT(*) as row_count,
  COUNT(DISTINCT column_name) as unique_count,
  COUNT(*) - COUNT(column_name) as null_count
FROM table_name
SAMPLE (10000 ROWS);

Profiling can be expensive on large tables. Use sampling and size limits appropriately. Run profiling separately from metadata ingestion on different schedules. Consider profiling only critical tables.

• Profiling Guide
• Dataset Profile