Intermittent Athena Query Source Failures

Athena query-based ingestion source experiences intermittent failures, where some ingestion runs succeed while others fail without clear error messages or consistent patterns.

Note: For DataHub Cloud customers, infrastructure-level changes are managed by DataHub Support.

1. Review Athena ingestion configuration:

source:
  type: athena
  config:
    aws_region: "us-east-1"
    work_group: "primary"
    # Query history settings
    query_log_delay: 15  # minutes to wait for query logs
    query_history_lookback_days: 7

2. Common causes of intermittent failures:

   • Athena query API rate limits being hit
   • Timeout waiting for query execution results
   • Temporary AWS service issues
   • Insufficient query execution permissions

3. Configure retry and timeout settings:

source:
  type: athena
  config:
    # Add resilience settings
    max_retry_count: 5
    retry_backoff_multiplier: 2
    query_timeout: 300  # seconds

4. Troubleshoot intermittent issues:

   • Review Athena CloudWatch metrics during failure periods
   • Check AWS Health Dashboard for service issues
   • Examine ingestion logs to identify patterns (time of day, query volume)
   • Consider increasing query_log_delay if queries take longer to appear

5. Monitor Athena API limits:

   • AWS has API rate limits for Athena query operations
   • Implement exponential backoff in ingestion
   • Contact AWS support if consistent throttling occurs

Test configurations in a development environment before applying to production. Always backup configurations before making changes.

• DataHub Documentation
• Athena Source

• 5362, 5750