Scheduled Ingestion Not Running

Ingestion pipelines configured with schedules through the UI are not running at the expected times, or run status shows as pending indefinitely. This commonly occurs due to executor pod issues, cron expression errors, or insufficient resources for the DataHub Actions container.

You Might Be Asking:

• Why isn't my scheduled ingestion running?
• How do I check the status of scheduled ingestions?
• What cron format should I use?

1. Verify executor pod is running:

# Check DataHub actions/executor pod
kubectl get pods | grep datahub-actions

# Check logs for execution
kubectl logs -f deployment/datahub-actions | grep "ingestion"

# Check for executor errors
kubectl logs deployment/datahub-actions | grep -i error

2. Validate cron schedule format:

# Standard cron format: minute hour day month day-of-week
0 2 * * *      # Daily at 2 AM
0 */6 * * *    # Every 6 hours
30 8 * * 1-5   # Weekdays at 8:30 AM
0 0 1 * *      # Monthly on the 1st

# Test your cron expression at https://crontab.guru/

3. Check ingestion execution history:

query {
  listIngestionSources(
    start: 0
    count: 20
  ) {
    ingestionSources {
      urn
      name
      schedule {
        interval
        timezone
      }
      executions(start: 0, count: 10) {
        executionRequests {
          result {
            status
            startTimeMs
            durationMs
          }
        }
      }
    }
  }
}

4. Ensure sufficient resources for executor:

# In Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: datahub-actions
spec:
  template:
    spec:
      containers:
      - name: datahub-actions
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"

5. Manually trigger ingestion to test:

mutation {
  createIngestionExecutionRequest(
    input: {
      ingestionSourceUrn: "urn:li:dataHubIngestionSource:your-source-id"
    }
  )
}

Scheduled ingestion requires DataHub Actions (executor) to be deployed and running. For high-frequency ingestion (< 15 minutes), consider using external orchestration tools like Airflow.

• UI Ingestion Guide
• DataHub Actions