Stale Ingestion 'Last Run' Timestamps Caused by Overloaded Remote Executor Schedules – DataHub Customer Support Portal

Area: Ingestion Issues

Sub-Area: Remote Executor / Ingestion Run Reporting

Issue

When multiple ingestion sources are scheduled to run simultaneously on a remote executor, the executor can become CPU-constrained at startup time, causing a capability check to time out. This timeout causes the executor to skip reporting the run result back to DataHub GMS, so the "Last Run" timestamp in the Manage Data Sources UI becomes stale and the Run History page appears incomplete — even though the underlying ingestion jobs completed successfully and asset-level metadata was updated correctly. The discrepancy between the stale "Last Run" column and the accurate "Synced: X hours ago" status shown on individual dataset pages is a direct symptom of this condition.

Error Messages

[datahub.ingestion.run.sink_callback] sink wrote workunit urn:li:dataset:(...) — present in executor logs, confirming ingestion ran, but run result is never reported back to GMS
Capability check timeout: datahub ingest run --help exceeds the 10-second limit and is silently skipped

You Might Be Asking

Why does the dataset page show "Synced: 10 hours ago" but the Data Sources list shows "Last Run: 7 days ago"?
Why is my ingestion Run History missing recent runs even though ingestion logs confirm successful execution?
Why did upgrading the Remote Executor not fix the stale Last Run timestamp?
Why does my remote executor intermittently fail to report run results back to DataHub?
Why do all my ingestion sources show multi-day gaps in run history when they are scheduled nightly?

Root Cause

After each ingestion run completes, the remote executor performs a separate step: it reports the run result (status and timestamp) back to DataHub so the result appears in Data Sources → Last Run and Run History. Before reporting, the executor runs a quick capability check — effectively datahub ingest run --help — with a hard 10-second timeout. When many ingestion sources are scheduled to start at the same time (e.g., all sources set to 00:00), multiple Python virtual environments must boot simultaneously and import the datahub package in parallel. This creates CPU contention on the executor pod. If the capability check exceeds 10 seconds due to CPU starvation, the executor incorrectly assumes run-result reporting is unsupported and silently skips it — even though the ingestion run itself succeeded and metadata reached DataHub normally.

This is a load-driven issue, not a version mismatch. The "Synced" timestamp on the asset page remains accurate because it is populated via a different code path (entity-level metadata written during the run itself), which is unaffected by the capability check timeout.

Solution

Apply one or more of the following mitigations. Implementing options 1 and 2 together is the most reliable resolution.

Stagger ingestion schedules (highest-impact fix).
Distribute your ingestion source schedules across a time window rather than starting all sources at the same time. Even a 10–15 minute offset between sources significantly reduces concurrent executor load and prevents the capability check from timing out.

Example: instead of scheduling all sources at 00:00, spread them as follows:
```
# Example cron schedule staggering for multiple ingestion sources
Source A: 0 0 * * *      # 00:00
Source B: 15 0 * * *     # 00:15
Source C: 30 0 * * *     # 00:30
Source D: 45 0 * * *     # 00:45
Source E: 0 1 * * *      # 01:00
```
Update each source's schedule via Manage Data Sources → Edit → Schedule in the DataHub UI, or via the DataHub API/CLI.
Reduce remote executor concurrency (worker/pool size).
Lower the number of ingestion tasks the executor pool runs simultaneously. Fewer concurrent cold-starts means the capability check completes within the 10-second window. Update the concurrency setting in your executor deployment configuration (Helm values or CloudFormation template):
```
# Example: Helm values for remote executor deployment
remoteExecutor:
  worker:
    poolSize: 2   # Reduce from default (e.g., 5 or more) to limit concurrency
```
Adjust the value based on your workload size and available CPU. A lower pool size reduces simultaneous Python environment boot load.
Increase CPU resources allocated to the remote executor pod.
The capability check is slow primarily because a fresh Python virtual environment must boot and import the datahub package, which is a heavy operation. Allocating more CPU to the executor pod (and ensuring CPU limits are not being throttled during peak ingestion windows) allows the check to complete within the timeout.
```
# Example: Kubernetes resource request/limit increase for remote executor
resources:
  requests:
    cpu: "2"
    memory: "4Gi"
  limits:
    cpu: "4"
    memory: "8Gi"
```
Tune these values based on the number of concurrent sources and your cluster capacity.

After applying one or more of the above changes, monitor the Data Sources → Run History page over the following days to confirm that "Last Run" timestamps are updating consistently and that no new gaps appear in the run history.

Verification

To confirm the fix is working, check that:

The "Last Run" timestamp in the Manage Data Sources list matches the expected most-recent run time.
The Run History page for each affected source shows no unexpected gaps.
The "Synced: X hours ago" value on the dataset detail page and the "Last Run" value in the source list now agree.
Executor logs no longer show the capability check timing out around the scheduled ingestion window.

Additional Notes

This issue is entirely load-driven. Upgrading the Remote Executor image version alone will not resolve the stale timestamps if schedules remain concentrated at the same time.
The "Synced" status shown on individual dataset/asset pages is sourced from entity-level metadata written during the ingestion run itself and is not affected by the capability check. It will appear correct even when "Last Run" in the Data Sources list is stale — this discrepancy is the key diagnostic signal for this issue.
Analysis of executor logs may reveal that a significant percentage of nightly runs (potentially 30–40% or more in heavily loaded environments) are affected when all sources share the same schedule. Each affected run will have a corresponding log entry confirming successful data writes, but no corresponding run result reported to GMS.
This behavior has been confirmed on Remote Executor deployments using DataHub Cloud. The 10-second capability check timeout is a fixed internal limit in the executor; the workarounds above address the load conditions that cause it to be exceeded.
Ensure your Remote Executor image version is kept in sync with your DataHub instance version (e.g., if your DataHub instance is on v1.0.2, use the v1.0.2-cloud RE image). While a version mismatch is not the root cause of this specific issue, running mismatched versions can introduce additional instability in run reporting.