DataHub Actions Framework: Offset Commit Behavior and Replay Risk on Ungraceful Shutdown – DataHub Customer Support Portal

```html

Area: Best Practices

Sub-Area: Actions Framework / Event Source Configuration

Issue

When using the DataHub Actions Framework with the DataHub Cloud Event Source, consumer progress (offsets) is only committed back to DataHub on a graceful shutdown of the pipeline. There is no periodic checkpointing during normal operation. This means that if the Actions pipeline process is killed, crashes, or terminates ungracefully — for example, due to a container restart, OOM kill, or SIGTERM not being handled correctly — the pipeline will resume from the last successfully committed position, which could be the last time the process shut down cleanly. In long-running deployments, this can result in a significant amount of event replay on the next startup.

You Might Be Asking

Does the DataHub Actions Framework commit Kafka/event offsets periodically, or only when the pipeline shuts down?
If my Actions pipeline crashes after running for days or weeks, how far back will it replay events?
Is there a checkpointing mechanism for the DataHub Cloud Event Source similar to the Kafka Event Source's async commit?
What is the difference in offset commit behavior between the DataHub Cloud Event Source and the Kafka Event Source?

Solution

The answer depends on which event source your Actions pipeline is using. The two most common sources have fundamentally different commit behaviors:

1. DataHub Cloud Event Source (DataHub Cloud deployments)

The DataHub Cloud Event Source polls a managed events API and tracks its position in memory as events are acknowledged. This in-memory position is only written back to DataHub when the pipeline shuts down cleanly. There is currently no periodic checkpointing. Key implications:

On an ungraceful shutdown (crash, OOM kill, unhandled SIGTERM), the pipeline resumes from the position of the last clean shutdown — potentially replaying a large volume of events.
There is a known bug where stop() is not called correctly on SIGTERM or inside containerized environments, meaning even intended restarts may not produce a clean shutdown. See the upstream issue datahub-project/datahub#19402 for details.
Periodic checkpointing for the Cloud Event Source is a recognized limitation and is on the roadmap for improvement, but as of the time of writing it is not yet implemented.

Mitigation strategies for the Cloud Event Source:

Ensure graceful shutdown wherever possible. Configure your container orchestrator (e.g., Kubernetes) to send SIGTERM and allow sufficient terminationGracePeriodSeconds for the process to shut down cleanly before SIGKILL is issued.
```
# Example Kubernetes pod spec snippet
spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: datahub-actions
      # ...
```
Design your Action handlers to be idempotent. Since event replay cannot be fully prevented, ensure that processing the same event more than once does not produce incorrect or duplicate side effects in downstream systems.
Monitor pipeline restarts. Alert on unexpected restarts so you are aware when replay may be occurring and can assess the impact.
Open or upvote a feature request with DataHub support if periodic checkpointing for the Cloud Event Source is important to your use case. Providing concrete business context helps prioritize the work.

2. Kafka Event Source (self-hosted or Kafka-backed deployments)

The Kafka Event Source has a more robust commit model and supports periodic async offset commits by default. This source is not subject to the same week-long replay risk as the Cloud Event Source.

Default behavior (async commits, recommended): After each event is successfully processed, the offset is stored locally via store_offsets(). A background thread commits all stored offsets to Kafka on a configurable interval (default: 10 seconds). On an ungraceful crash, you will only replay at most the last ~10 seconds of events.
Synchronous per-event commits (maximum durability): Commits are written to Kafka after every single event. Near-zero replay risk, but approximately 25× lower throughput than async mode.

Example Kafka Event Source configuration showing commit strategy options:

# datahub-actions pipeline config (YAML)
source:
  type: "kafka"
  config:
    connection:
      bootstrap: ":9092"
    # --- Async commits (default, recommended) ---
    async_commit_enabled: true
    async_commit_interval: 10000   # milliseconds; reduce for tighter guarantees

    # --- OR: Synchronous per-event commits (high durability, lower throughput) ---
    # async_commit_enabled: false

Commit strategy comparison for the Kafka Event Source:

Mode	Config	Approximate Throughput	Max Replay on Crash
Async (default)	`async_commit_enabled: true`, `async_commit_interval: 10000`	~8,200 events/sec	~10 seconds of events
Tuned async	`async_commit_enabled: true`, `async_commit_interval: 5000`	~8,200 events/sec	~5 seconds of events
Sync per-event	`async_commit_enabled: false`	~326 events/sec	Near-zero

Additional Notes

The distinction between the two event sources is critical. If you are on DataHub Cloud and using the managed DataHub Cloud Event Source, the Kafka Event Source commit settings described above do not apply — those settings are only relevant for self-managed Kafka-backed pipelines using the Kafka Event Source. All built-in DataHub Actions (such as ingestion actions) are designed to be idempotent, so replayed events should not cause data corruption, but custom action handlers must also be written with idempotency in mind to be safe under replay conditions. The SIGTERM handling bug affecting the Cloud Event Source in containerized environments (datahub-project/datahub#19402) means that even graceful restarts may not reliably commit progress in some deployment configurations — check the linked GitHub issue for the current fix status before relying on graceful shutdown as a sole mitigation.