Snowflake Profiling Showing Incorrect Numbers

The Snowflake profiling ingestion recipe displayed different row count and statistics numbers than what was actually ingested into DataHub. This discrepancy between the reported profiling metrics during ingestion and the actual data stored in DataHub caused confusion about data quality and completeness.

You Might Be Asking:

• Why don't the profiling numbers match what's in DataHub?
• How accurate is Snowflake profiling?
• What metrics should I trust?

The Snowflake profiling connector has been fixed to accurately report and ingest profiling statistics. The issue was related to how the connector aggregated results from Snowflake's information schema.

Verify profiling accuracy by checking both the ingestion logs and the DataHub UI:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    account_id: "your_account"
    warehouse: "your_warehouse"

    profiling:
      enabled: true
      profile_table_level_only: false

      # Ensure accurate row counting
      max_workers: 5
      profile_if_updated_since_days: 1

Fixed in version v0.3.11. After upgrading, rerun profiling ingestion to get accurate statistics. The fix ensures that row counts, null percentages, and other profiling metrics are correctly calculated and displayed.

• Snowflake Ingestion Source
• Dataset Profiling