Redash Ingestion Fails or Times Out Due to Long Run Duration and Executor Restarts – DataHub Customer Support Portal

```html

Area: Ingestion Issues

Sub-Area: Redash Connector Performance and Executor Stability

Issue

Redash ingestion runs with large numbers of queries and dashboards (for example, tens of thousands of queries) can take several hours to complete when using default connector settings (page_size: 25, parallelism: 1). In environments where the ingestion executor coordinator is subject to periodic restarts — caused by liveness probe timeouts under heavy in-process workloads — any ingestion run that exceeds the restart interval will be killed mid-flight and reported as ABORTED. Successive retries are also killed, resulting in metadata going stale without a clear error signal. The same restart window can affect other long-running ingestion sources (such as BigQuery) running in the same executor. Reducing the Redash run duration via page_size and parallelism tuning is an effective workaround that allows runs to complete well within the restart window.

Error Messages

Ingestion was aborted due to worker pod eviction, crash, or restart.
Killing Container acryl-datahub-executor-coordinator failed liveness probe, will be restarted
Liveness probe failed: Get "http://<pod-ip>:9004/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Run time of job "SweeperJob.run" was missed by <duration>
Run time of job "ExecutionRequestManager.refresh_execution_requests" was missed by <duration>

You Might Be Asking

Why does my Redash ingestion always show as ABORTED and never complete?
Why does Redash ingestion abort at roughly the same time every day, even on retries?
My Redash ingestion used to complete but now fails every run — what changed?
How can I make Redash ingestion faster without changing my Redash infrastructure significantly?
Does executor restart affect other ingestion sources besides Redash?
Why does the last completed ingestion report show "failures": [] even though runs are being aborted?

Solution

Step 1: Confirm the Root Cause

Check whether the pattern matches an executor restart issue rather than a connector-level failure:

Multiple ingestion attempts all abort at approximately the same elapsed time (e.g., every ~2–3 hours), regardless of how much data has been processed.
Multiple long-running ingestion sources (e.g., Redash and BigQuery) abort at exactly the same UTC timestamp, indicating a single restart event kills all in-flight runs simultaneously.
The last successfully completed run's report shows "failures": [] — no connector-level errors are present.
Ingestion runs that complete in minutes are unaffected; only runs exceeding the restart interval are killed.
The DataHub ingestion UI shows status ABORTED (not FAILED) for the affected runs.

Step 2: Reduce Redash Run Duration via Recipe Tuning

The fastest workaround for Redash specifically is to increase page_size and parallelism in your ingestion recipe. This reduces a multi-hour run to well under one hour for most large Redash instances, allowing it to complete between executor restarts.

In the DataHub UI, navigate to Ingestion and open your Redash ingestion source.
Click Edit to open the recipe editor.
Under the config block, add the following two parameters alongside your existing settings (connect_uri, api_key, etc.):

source:
  type: redash
  config:
    connect_uri: "https://<your-redash-instance>/"
    api_key: "${REDASH_API_KEY}"
    parse_table_names_from_sql: true
    page_size: 100
    parallelism: 8

Or in JSON recipe format:

{
  "source": {
    "type": "redash",
    "config": {
      "connect_uri": "https://<your-redash-instance>/",
      "api_key": "${REDASH_API_KEY}",
      "parse_table_names_from_sql": true,
      "page_size": 100,
      "parallelism": 8
    }
  }
}

page_size: 100 — Fetches queries in larger batches (default is 25), reducing the number of API round-trips significantly.
parallelism: 8 — Fetches up to 8 queries concurrently instead of sequentially (default is 1). This is the primary driver of the speedup.

Save the recipe and trigger a manual run to verify the new duration. A run that previously took 4+ hours should complete in approximately 1 hour or less depending on instance size.

Step 3: Assess Parallelism Load on Your Redash Instance

Before applying parallelism: 8, confirm with your Redash infrastructure team that the additional concurrent API requests are acceptable. Key factors to communicate:

The parallel requests only occur during the ingestion run window (typically once per day on your scheduled cadence).
With parallelism: 8, the burst is short-lived — for a large instance it lasts under one hour instead of several hours of sequential requests at lower throughput.
If your Redash instance cannot handle 8 concurrent requests, use a lower value such as parallelism: 4 and increase page_size further to compensate.

Step 4: Address the Underlying Executor Restart Issue (DataHub Cloud)

If you are on DataHub Cloud and the run-duration workaround does not fully resolve the issue (for example, other long-running sources such as BigQuery continue to abort), the underlying executor coordinator stability must be addressed at the infrastructure level. This is not a customer-configurable setting.

Contact DataHub Support and provide: the affected ingestion source URNs, run history showing correlated abort timestamps across sources, and the approximate run duration of each affected source.
Support will investigate the executor coordinator's liveness probe behavior, APScheduler job scheduling latency, and memory headroom. Common infrastructure-level remediations include increasing executor memory limits and adjusting liveness probe parameters.
The recipe-level tuning in Step 2 remains valuable even after an executor fix, as shorter runs are inherently more resilient and reduce load on the coordinator.

Step 5: Verify Resolution

After applying the recipe changes and any infrastructure fixes, confirm resolution by checking the following:

The ingestion run completes with status SUCCESS rather than ABORTED.
Run duration has decreased substantially compared to pre-tuning baselines.
Subsequent scheduled runs also complete successfully over multiple consecutive days.
Other long-running ingestion sources in the same executor (e.g., BigQuery) are no longer aborting at the same timestamps as Redash runs.

Additional Notes

The parallelism setting controls concurrent HTTP requests to the Redash API. Higher values yield faster runs but place more load on your Redash server. A value of 8 is a reasonable starting point for most production Redash instances; reduce it if you observe API rate limiting or elevated error rates on the Redash side. The page_size setting affects how many query objects are fetched per API page; the maximum effective value depends on your Redash version and server configuration, but 100 is broadly supported.

The executor restart behavior described here is caused by event-loop and thread starvation in the ingestion executor coordinator when multiple long-running in-process ingestion jobs are active simultaneously. This can cause APScheduler background jobs (including the health-check endpoint) to miss their scheduled windows, causing the Kubernetes liveness probe to time out and restart the container. All in-flight ingestion runs are killed when this happens. The ABORTED status is assigned by the sweeper after a configurable threshold (default 3600 seconds), so there is a delay between the actual kill event and the status update visible in the UI. Because the last completed run report shows "failures": [], this failure mode produces no error-level alert and can go undetected unless run history is actively monitored.

These settings apply to the Redash ingestion connector specifically. Other ingestion sources that experience long run durations due to large data volumes (e.g., BigQuery with many datasets) may require infrastructure-level fixes rather than recipe tuning, as those connectors do not have equivalent parallelism controls that reduce end-to-end wall-clock time proportionally.

Tested with Redash source connector on DataHub Cloud. The page_size and parallelism parameters are available in recent versions of the acryl-datahub Redash plugin. Verify that your pinned CLI/plugin version supports these parameters if you are using a custom wheel or non-standard version.