Stateful Ingestion Configuration and Checkpointing Issues

Stateful ingestion configuration fails or does not properly checkpoint progress, leading to duplicate data ingestion or inability to resume from where the previous run left off. This can cause performance issues and data inconsistencies.

Note: For DataHub Cloud customers, infrastructure-level changes (Kubernetes, AWS resources, operating system configurations) are managed by DataHub Support. The solutions below focus on configuration changes that customers can implement.

1. Enable stateful ingestion in your recipe:

source:
  config:
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true

2. Verify checkpoint storage:

   • Ensure the DataHub GMS endpoint is accessible from the ingestion executor
   • Check that the ingestion source user/token has permissions to write checkpoints
   • Confirm checkpoint state is being persisted by reviewing subsequent run logs

3. Troubleshoot checkpoint failures:

   • Clear existing checkpoint if corrupted: Use the DataHub UI or CLI to reset the checkpoint
   • Check for clock skew between ingestion executor and DataHub server
   • Review logs for checkpoint serialization errors

4. Verify state is being used:

   • Compare first run vs second run - second run should be faster and skip unchanged entities
   • Check ingestion logs for messages about loaded checkpoint state
   • Review the number of entities processed in subsequent runs

Always backup your DataHub configuration before making changes. Test solutions in a non-production environment when possible.

• Ingestion Sources

• 5763, 5643