False Positive Data Quality Check Alerts

Users receive intermittent "false positive" alerts from DataHub data quality checks (e.g., dbt tests, assertions, custom checks) where checks fail temporarily but the underlying data is actually correct. These false alarms can persist for a period (e.g., one week) and then resolve without clear explanation, making it difficult to trust the monitoring system.

You Might Be Asking:

• Why am I getting false positive alerts from my data quality checks?
• How can I diagnose the root cause of intermittent check failures?
• What logs should I preserve to troubleshoot false positives?
• How can I reduce false positive rates?

Diagnose false positives by preserving detailed execution logs, implementing proper thresholds, and understanding the timing and caching of data quality checks.

Common Causes of False Positives:

1. Timing and Freshness Issues: - Check runs before data is available - Time zone mismatches - Delayed data pipeline completion

Example:

# Problem: Check runs too early
assertion:
  schedule: "0 6 * * *"  # 6 AM
  # But data arrives at 6:30 AM

# Solution: Adjust schedule
assertion:
  schedule: "0 7 * * *"  # 7 AM
  # Or add freshness buffer

2. Threshold Sensitivity: - Too strict thresholds - No tolerance for normal variance

Example:

# Problem: Too strict
assertion:
  type: VOLUME
  min_rows: 1000
  max_rows: 1000  # Exact match required

# Solution: Add buffer
assertion:
  type: VOLUME
  min_rows: 950   # 5% tolerance
  max_rows: 1050

3. Caching and Staleness: - Cached results from previous failures - Stale metadata in DataHub

4. Transient Source Issues: - Temporary database performance issues - Network timeouts - Query timeouts

Diagnostic Approach:

Step 1: Preserve Execution Logs

Always save detailed logs from check executions:

# For dbt tests
dbt test --store-failures --output-path logs/

# For custom checks
assertion:
  config:
    log_level: DEBUG
    save_results: true
    results_path: "s3://bucket/assertions-results/"

Step 2: Implement Detailed Logging

# Custom assertion with detailed logging
import logging
from datetime import datetime

logger = logging.getLogger(__name__)

def run_custom_check(dataset_urn):
    """Run check with detailed logging"""

    timestamp = datetime.utcnow().isoformat()
    logger.info(f"Starting check for {dataset_urn} at {timestamp}")

    try:
        # Run the actual check
        result = execute_quality_check()

        logger.info(f"Check result: {result}")
        logger.info(f"Row count: {result['row_count']}")
        logger.info(f"Null count: {result['null_count']}")
        logger.info(f"Execution time: {result['duration_ms']}ms")

        # Save to persistent storage
        save_to_s3(f"checks/{dataset_urn}/{timestamp}.json", result)

        return result

    except Exception as e:
        logger.error(f"Check failed: {str(e)}", exc_info=True)
        raise

Step 3: Analyze Failure Patterns

# Analyze historical check results
def analyze_false_positives(check_history):
    """Identify patterns in false positives"""

    failures = [r for r in check_history if r['status'] == 'FAIL']

    # Check for timing patterns
    failure_hours = [r['timestamp'].hour for r in failures]
    print(f"Failures by hour: {Counter(failure_hours)}")

    # Check for transient failures
    transient = [
        f for f in failures 
        if f['next_run_status'] == 'PASS'
    ]
    print(f"Transient failures: {len(transient)}/{len(failures)}")

    # Check for threshold breaches
    near_threshold = [
        f for f in failures
        if abs(f['actual_value'] - f['threshold']) < f['threshold'] * 0.05
    ]
    print(f"Near-threshold failures: {len(near_threshold)}")

Best Practices to Reduce False Positives:

1. Implement Retries:

assertion:
  retry_policy:
    max_retries: 3
    retry_delay_seconds: 300  # 5 minutes
    backoff_multiplier: 2

2. Use Appropriate Thresholds:

# Use percentage-based thresholds
assertion:
  type: VOLUME
  # Instead of absolute values
  min_rows_percent_change: -10  # Allow 10% decrease
  max_rows_percent_change: 20   # Allow 20% increase

3. Add Warm-up Periods:

# Don't alert immediately
assertion:
  alert_policy:
    consecutive_failures_required: 2  # Alert after 2 failures
    suppress_duration_minutes: 60     # No alerts for first hour

4. Validate Check Logic:

-- Test the check SQL manually
SELECT 
  COUNT(*) as row_count,
  COUNT(*) BETWEEN 950 AND 1050 as passes_check,
  CURRENT_TIMESTAMP as check_time
FROM my_table
WHERE date = CURRENT_DATE;

5. Monitor Data Source Health: - Check for upstream pipeline delays - Monitor source system performance - Verify data availability timing

Troubleshooting Checklist:

• [ ] Preserve execution logs and results files
• [ ] Check execution timing relative to data availability
• [ ] Review threshold values for reasonableness
• [ ] Verify source data directly at time of failure
• [ ] Check for network or connection issues
• [ ] Review recent changes to source systems
• [ ] Compare with historical success/failure patterns
• [ ] Validate check logic independently

When to Contact Support:

If false positives persist after troubleshooting: 1. Provide detailed logs from multiple failure instances 2. Include timestamps of failures and subsequent successes 3. Share assertion/check configuration 4. Describe data source characteristics and timing 5. Provide results.json or equivalent execution artifacts

• False positives undermine trust in data quality monitoring
• Root cause analysis requires detailed historical data
• Most false positives are configuration issues, not DataHub bugs
• Proper threshold setting is critical for reliable monitoring
• Consider implementing gradual rollout for new checks

• Assertions
• dbt Integration

• 4469