Stateful Ingestion Causing Duplicate or Missing Assets

When stateful ingestion is enabled, assets may appear duplicated, go missing, or fail to update properly. This typically happens when the state is corrupted, the ingestion source changes significantly, or when multiple ingestion jobs run concurrently with overlapping scopes.

Common Error Messages:

• Failed to retrieve previous checkpoint
• State file corrupted or unreadable
• Checkpoint version mismatch

You Might Be Asking:

• What is stateful ingestion and when should I use it?
• How do I reset the stateful ingestion state?
• Why are my assets disappearing after enabling stateful ingestion?

1. To reset stateful ingestion state, remove the checkpoint:

# In your recipe, add:
source:
  config:
    stateful_ingestion:
      enabled: true
      remove_stale_metadata: true
      # To force a full refresh, delete the state

2. Delete the state file manually:

# For local file-based state
rm -rf /tmp/datahub/ingestion_state/

# For DataHub backend state
# Use the GraphQL API to clear the checkpoint

3. Configure stateful ingestion properly for your use case:

Example for Snowflake. This can be adapted for other data sources/connectors.

source:
  type: snowflake
  config:
    stateful_ingestion:
      enabled: true
      # Remove assets that no longer exist in source
      remove_stale_metadata: true
      # Fail if state cannot be retrieved
      fail_safe_threshold: 100.0

4. Avoid concurrent ingestion runs with overlapping scopes:
   • Use different identifiers for different ingestion pipelines
   • Schedule ingestion jobs to avoid overlap
   • Use platform_instance to separate environments

Stateful ingestion is recommended for production deployments to handle asset deletions and reduce ingestion time. However, for initial setup or major source changes, consider running without stateful ingestion first.

• Stateful Ingestion Guide