Stale RemoteExecutorStatus Record Blocking Ingestion: Resolution via Direct Elasticsearch Deletion – DataHub Customer Support Portal

```html

Area: Deployment Issues

Sub-Area: Executor Coordinator / Ingestion Pipeline Failures

Issue

A stale or malformed RemoteExecutorStatus record persisted in the metadata store can block all ingestion sources assigned to an executor pool (commonly the default pool). The record is missing the required executorPoolId field — typically introduced during a version upgrade that added multi-pool executor support — making it impossible to deserialize. Because the executor coordinator's sweeper can only act on records it can successfully deserialize, the corrupt record is silently dropped from its working set and never scheduled for cleanup. The result is a self-perpetuating failure: scheduled ingestion runs transition from Pending to Running but no ingestion pods are ever spawned, runs accumulate in a zombie Running state, and the coordinator emits a deserialization error every ~60 seconds indefinitely. Standard API and CLI soft-delete operations are insufficient on their own because a copy of the corrupt document may persist in the Elasticsearch index even after the primary metadata store record is removed.

Error Messages

ERROR datahub_executor.common.graph - Failed to load RemoteExecutorStatus record with urn urn:li:dataHubRemoteExecutor:<executor-urn>: com.linkedin.pegasus2avro.executor.RemoteExecutorStatus is missing required field: executorPoolId
Sweeper: plan execution finished: 0 of 0 actions succeeded.

You Might Be Asking

Why do my ingestion runs show as Running but no pods are created?
Why does the executor coordinator log a deserialization error every minute against the same URN?
Why did running datahub delete --hard or the bootstrap purge job not stop the error?
How do I permanently remove a corrupt RemoteExecutorStatus record that cannot be deleted through the standard UI or API?
What causes RemoteExecutorStatus is missing required field: executorPoolId?

Root Cause

The executorPoolId field in RemoteExecutorStatus.pdl is declared as a required, non-optional string. When an executor pod running an older image version (pre-dating multi-pool support) emits a heartbeat status record without this field and is subsequently stopped or replaced before re-registering, the persisted record becomes permanently schema-incompatible. The coordinator's get_remote_executors() method catches the deserialization exception and logs it, but silently omits the URN from its return value. The sweeper's _get_remote_executor_actions() function can only schedule cleanup actions for records that were successfully returned — so the corrupt record is never targeted for deletion and persists indefinitely. Additionally, a prior soft-delete or partial cleanup attempt may leave a residual document in the Elasticsearch index that continues to be read by the coordinator even after the primary aspect table record is removed.

Solution

Resolution requires a three-stage approach. Work through each stage in order, verifying after each step whether the coordinator error has stopped before proceeding to the next.

Stage 1 — Run the Bootstrap Purge Job (Recommended First Step)
Set the following environment variables in your datahubSystemUpdate (upgrade job) configuration and re-run the upgrade job. This purge step clears legacy executor records from both the primary store and Elasticsearch when it functions correctly.
```
datahubSystemUpdate:
  extraEnvs:
    - name: BOOTSTRAP_SYSTEM_UPDATE_PURGE_LEGACY_EXECUTORS_ENABLED
      value: "true"
    - name: BOOTSTRAP_SYSTEM_UPDATE_PURGE_LEGACY_EXECUTORS_REPROCESS
      value: "true"
```
Because the system-update job has restartPolicy: Never, you must delete the old job and trigger a Helm upgrade to recreate it:
```
# Delete the existing completed job
kubectl delete job <release-name>-system-update

# Re-run via Helm upgrade
helm upgrade <release-name> <chart> -f values.yaml
```
Note: If systemUpdate.scaleDown.enabled is set, this will briefly scale GMS consumers to zero during the run. To avoid downtime, you can run the system-update image manually with the -u SystemUpdate flag and the two environment variables exported, which performs the same purge without touching live deployments.

After the job completes, restart the executor-coordinator pod and check whether the deserialization error has stopped. If the log is clean, flip BOOTSTRAP_SYSTEM_UPDATE_PURGE_LEGACY_EXECUTORS_REPROCESS back to false to prevent the purge from re-running on every subsequent restart.

If the error persists (the purge step may be a no-op when the record is already in a soft-deleted state), proceed to Stage 2.
Stage 2 — Hard Delete via DataHub CLI
Identify the affected URN from the coordinator error log (it will match the pattern urn:li:dataHubRemoteExecutor:<executor-id>) and issue a hard delete:
```
datahub delete \
  --urn "urn:li:dataHubRemoteExecutor:<your-executor-urn>" \
  --hard
```
Restart the executor-coordinator pod after the delete. If the error log is clean, the issue is resolved. If the hard delete reports 0 versioned rows and 0 timeseries aspect rows impacted and the error continues, the record exists only in Elasticsearch — proceed to Stage 3.
Stage 3 — Direct Elasticsearch Deletion (Final Resolution)
This stage is required when the primary metadata store is already clean but a residual document remains in the Elasticsearch index.

Step 3a — Identify the correct index:
```
GET _cat/indices?v | grep -i remoteexecutor
```
The index is typically named datahubremoteexecutorindex_v2, optionally prefixed if your deployment uses an index prefix.

Step 3b — Confirm the document is present:
```
GET <index-name>/_search
{
  "query": {
    "term": {
      "urn": "urn:li:dataHubRemoteExecutor:<your-executor-urn>"
    }
  }
}
```
If no results are returned, try "urn.keyword" instead of "urn" depending on your index mapping.

Step 3c — Delete the document by query:
```
POST <index-name>/_delete_by_query
{
  "query": {
    "term": {
      "urn": "urn:li:dataHubRemoteExecutor:<your-executor-urn>"
    }
  }
}
```
Or equivalently via curl from within a coordinator pod:
```
curl -X POST "http://<elasticsearch-host>:9200/<index-name>/_delete_by_query" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "term": {
        "urn": "urn:li:dataHubRemoteExecutor:<your-executor-urn>"
      }
    }
  }'
```
A successful response will include "deleted": 1, "failures": [].

Step 3d — Restart the executor-coordinator:
```
kubectl rollout restart deployment/<executor-coordinator-deployment-name>
```
On the next coordinator cycle (~60 seconds), the deserialization error should no longer appear in the logs.
Post-Resolution Cleanup
After confirming the coordinator log is clean, cancel any ingestion runs that were left in a zombie Running state via the DataHub UI and re-trigger them manually. These runs will not self-recover — they must be restarted explicitly.

Live executors will automatically re-register their status records (heartbeats) within seconds of restart, so no executor data is lost by removing the stale index document.

Additional Notes

This issue is most commonly triggered by a version upgrade that introduced executorPoolId as a required field in RemoteExecutorStatus, when an executor pod running the older image left a heartbeat record in the store before being stopped. The stored record predates the schema change and cannot be deserialized by the newer coordinator.
The bootstrap purge job (PurgeLegacyExecutors step) is the intended automated fix, but it only targets live (non-soft-deleted) records. If a prior soft-delete or API delete attempt has already marked the record as deleted, the purge step will be a no-op and direct Elasticsearch deletion (Stage 3) is required.
This behavior has been identified as a product-level gap: the sweeper cannot self-heal records it cannot deserialize, and no schema migration was applied at upgrade time to backfill the executorPoolId field into pre-existing records. A future release is expected to introduce automated remediation for this scenario.
Ingestion sources on other executor pools (not the affected pool) are not impacted by this issue.
The Elasticsearch deletion is low-risk: the datahubremoteexecutorindex_v2 index holds only heartbeat documents. Live executors re-register automatically on startup.
After applying the Stage 1 fix, set BOOTSTRAP_SYSTEM_UPDATE_PURGE_LEGACY_EXECUTORS_REPROCESS back to false (or remove the env var) to prevent unnecessary re-runs on future restarts.
Confirmed resolved on DataHub Acryl v0.3.17.x on-premise (Kubernetes) deployments.