Monitoring DataHub Remote Executor Ingestion with Prometheus Metrics – DataHub Customer Support Portal

Area: Observability Issues

Sub-Area: Remote Executor Monitoring & Alerting

Issue

When ingestion recipes are triggered via an external orchestrator such as Apache Airflow, the orchestrator may report a successful job trigger even if the underlying DataHub ingestion recipe fails internally. This results in silent ingestion failures that go undetected unless logs are manually inspected. Organizations need a reliable way to detect these failures programmatically by integrating DataHub Remote Executor Prometheus metrics with monitoring tools such as Datadog or Grafana, and by setting up alerts on key failure metrics.

You Might Be Asking

How do I monitor DataHub ingestion success and failure using Prometheus?
What Prometheus metrics does the DataHub Remote Executor expose?
How do I set up Datadog or Grafana alerts for ingestion failures in DataHub?
How can I detect silent ingestion failures when using an external orchestrator like Airflow?
How many Prometheus metrics does the DataHub Remote Executor expose, and what is the cardinality?

Solution

Verify the Prometheus metrics endpoint is accessible.
The DataHub Remote Executor exposes a Prometheus-compatible metrics endpoint. Confirm it is reachable by running:
```
curl http://<your-executor-host>:9087/metrics
```
This will return a full list of all currently exposed metrics and their current values.
Configure your monitoring tool to scrape the endpoint.
Add the executor's metrics endpoint as a scrape target in your Prometheus configuration, then visualize in Datadog or Grafana. For Prometheus, add the following to your prometheus.yml:
```
scrape_configs:
  - job_name: 'datahub-remote-executor'
    static_configs:
      - targets:
          - '<your-executor-host>:9087'
```
See the official monitoring documentation for full Prometheus configuration options.
Set up alerts on the key ingestion failure metrics.
The following metrics are the most critical for detecting silent ingestion failures when using an external orchestrator:
- datahub_executor_worker_ingestion_errors{pool_name} — Directly tracks the number of ingestion failures. Alert when this counter increments.
- datahub_executor_ingestion_handler_errors{exception} — Broader ingestion handler failures, labeled by exception type.
- datahub_executor_scheduler_submission_errors{pool_name,embedded,exception} — Job submission failures before execution begins.
Example Prometheus alerting rule for ingestion errors:
```
groups:
  - name: datahub_ingestion_alerts
    rules:
      - alert: DataHubIngestionFailure
        expr: increase(datahub_executor_worker_ingestion_errors[5m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "DataHub ingestion failure detected"
          description: "One or more ingestion jobs have failed on pool {{ $labels.pool_name }}."
```
Review the complete list of available metrics.
The Remote Executor exposes metrics across the following categories:

Ingestion Job Metrics (Most Critical for Failure Detection)
- datahub_executor_worker_ingestion_requests{pool_name} — Number of ingestion requests received/processed
- datahub_executor_worker_ingestion_errors{pool_name} — Number of ingestion failures
- datahub_executor_worker_ingestion_visibility_timeout{pool_name} — Jobs skipped due to SQS visibility timeout
- datahub_executor_worker_ingestion_cancel_requests — Cancellation requests received
- datahub_executor_worker_ingestion_fetch_signal_requests — Count/time spent fetching cancel signals
- datahub_executor_worker_ingestion_fetch_signal_errors{exception} — Errors fetching signals
- datahub_executor_worker_ingestion_fallback_fetch_requests — GMS-fallback fetch operations
Thread Pool & Capacity Metrics
- datahub_executor_worker_thread_pool_max_threads{thread_pool_name} — Maximum configured worker threads
- datahub_executor_worker_thread_pool_active_threads{thread_pool_name} — Currently active threads
- datahub_executor_worker_thread_pool_active_weight{thread_pool_name} — Total task weight currently executing
- datahub_executor_worker_thread_pool_max_weight{thread_pool_name} — Maximum task weight capacity
Memory Metrics
- datahub_executor_memory_usage_processgroup{kind} — Process group memory (RSS/VMS)
- datahub_executor_memory_usage_process{kind} — Main process memory (RSS/VMS)
- datahub_executor_memory_limit_processgroup{collector} — Memory limit (from cgroup/host)
- datahub_executor_memory_usage_host — Total host memory used
- datahub_executor_memory_limit_host — Total host memory available
CPU Metrics
- datahub_executor_cpu_usage_process{kind} — Process CPU time (user/system)
- datahub_executor_cpu_usage_processgroup{kind} — Process group CPU time
- datahub_executor_cpu_limit_processgroup{collector} — CPU limit (from cgroup/host)
- datahub_executor_cpu_usage_host — Total host CPU utilization (%)
- datahub_executor_cpu_limit_host — Total host CPU capacity (%)
Network I/O Metrics (Per Interface)
- datahub_executor_net_bytes_sent{interface} — Bytes transmitted
- datahub_executor_net_bytes_recv{interface} — Bytes received
- datahub_executor_net_packets_sent{interface} — Packets transmitted
- datahub_executor_net_packets_recv{interface} — Packets received
- datahub_executor_net_errors_in{interface} — Receive errors
- datahub_executor_net_errors_out{interface} — Transmit errors
- datahub_executor_net_dropin{interface} — Dropped inbound packets
- datahub_executor_net_dropout{interface} — Dropped outbound packets
Disk Metrics (Per Device/Mount Point)
- datahub_executor_disk_bytes_total{dev,mountpoint,fstype} — Total disk space
- datahub_executor_disk_bytes_used{dev,mountpoint,fstype} — Used disk space
- datahub_executor_disk_bytes_free{dev,mountpoint,fstype} — Free disk space
Coordinator-Specific Metrics — Ingestion Fetcher
- datahub_executor_ingestion_fetcher_requests — Time/count fetching execution requests
- datahub_executor_ingestion_fetcher_errors{exception} — Fetch errors by exception type
- datahub_executor_ingestion_fetcher_items_mapped — Items fetched and processed
- datahub_executor_ingestion_fetcher_items_errored{exception} — Items that failed to process
Coordinator-Specific Metrics — Scheduler
- datahub_executor_scheduler_ingestion_requests{pool_name} — Ingestion jobs scheduled
- datahub_executor_scheduler_submission_errors{pool_name,embedded,exception} — Job submission failures
- datahub_executor_scheduler_message_size_exceeded — Messages too large for the queue
Coordinator-Specific Metrics — Ingestion Handler
- datahub_executor_ingestion_handler_requests{pool_name,embedded} — Handler requests
- datahub_executor_ingestion_handler_errors{exception} — Handler errors
- datahub_executor_ingestion_kafka_mcl_events — Kafka MCL events received
- datahub_executor_ingestion_kafka_exec_events — Kafka execution events received
- datahub_executor_ingestion_emit_mcp_events — MCP events emitted
- datahub_executor_ingestion_emit_mcp_errors{exception} — MCP emission errors
Coordinator-Specific Metrics — Sweeper (Stuck Job Recovery)
- datahub_executor_sweeper_actions_planned{action} — Recovery actions scheduled
- datahub_executor_sweeper_actions_executed{action} — Recovery actions executed
- datahub_executor_sweeper_execution_requests{status} — Sweep requests by status
Assertion Metrics (Data Quality Monitoring)
- datahub_executor_worker_assertion_requests{pool_name} — Assertions processed
- datahub_executor_worker_assertion_errors{pool_name,exception} — Assertion failures
- datahub_executor_worker_assertion_evaluate_requests — Evaluation requests
- datahub_executor_worker_assertion_evaluate_errors{exception} — Evaluation errors
- datahub_executor_assertion_fetcher_requests — Fetch time/count
- datahub_executor_assertion_fetcher_errors{exception} — Fetch errors
- datahub_executor_assertion_fetcher_items_mapped — Items processed
- datahub_executor_assertion_fetcher_items_errored{exception} — Items failed
- datahub_executor_scheduler_assertion_requests{pool_name,embedded} — Assertions scheduled
Monitor Training Metrics
- datahub_executor_worker_monitor_training_requests{pool_name} — Training jobs processed
- datahub_executor_worker_monitor_training_errors{pool_name,exception} — Training failures
- datahub_executor_worker_monitor_training_executor_requests — Executor requests
- datahub_executor_worker_monitor_training_executor_errors{exception} — Executor errors
- datahub_executor_scheduler_monitor_training_requests{pool_name,embedded} — Training jobs scheduled
Monitor Bootstrap Metrics — Worker
- datahub_executor_worker_monitor_bootstrap_requests{pool_name} — Bootstrap requests processed
- datahub_executor_worker_monitor_bootstrap_execute_requests — Execution time
- datahub_executor_worker_monitor_bootstrap_executor_errors{exception} — Bootstrap failures
- datahub_executor_worker_monitor_bootstrap_skipped_stale{state} — Skipped non-executable monitors
Monitor Bootstrap Metrics — Fetcher
- datahub_executor_monitor_bootstrap_fetcher_submitted — Monitors submitted
- datahub_executor_monitor_bootstrap_fetcher_rejected — Monitors rejected
- datahub_executor_monitor_bootstrap_fetcher_errors{exception} — Fetch errors
- datahub_executor_monitor_bootstrap_fetcher_claim_errors{exception} — Claim errors
- datahub_executor_monitor_bootstrap_fetcher_items_mapped — Items parsed
- datahub_executor_monitor_bootstrap_fetcher_items_errored{exception} — Parse failures
- datahub_executor_monitor_bootstrap_fetcher_skipped_no_slots — Skipped due to capacity
- datahub_executor_monitor_bootstrap_fetcher_active_monitors — Currently active monitors
- datahub_executor_monitor_bootstrap_fetcher_available_slots — Available capacity slots
Monitor Bootstrap Metrics — Sweeper
- datahub_executor_monitor_bootstrap_sweeper_swept_count — Stuck monitors processed
- datahub_executor_monitor_bootstrap_sweeper_actions{target_state} — Actions