Metadata Tests Not Running Due to Batching Timeout in Large-Scale Environments – DataHub Customer Support Portal

```html

Area: Observability Issues

Sub-Area: Metadata Tests / BatchTestEngine Execution

Issue

In large-scale DataHub deployments with a high volume of entities (e.g., hundreds of thousands of datasets) and a significant number of configured metadata tests, the nightly metadata test evaluation job may consistently exceed its Kubernetes execution deadline and be terminated before it can complete. Because the BatchTestEngine persists results only after the full evaluation loop finishes, any job killed mid-run writes no results at all. The consequence is that the "Last computed" timestamp on every metadata test card in the UI freezes at the date of the last successful full run, making it appear as though tests have not executed for weeks or months — even though the CronJob itself is scheduling and launching correctly every night.

Error Messages

Job dh-metadata-tests-tpl-<id> DeadlineExceeded Job was active longer than specified deadline
CronJob dh-metadata-tests-tpl SawCompletedJob Saw completed job: dh-metadata-tests-tpl-<id>, condition: Failed
Pod <pod-name> Killing Stopping container metadata-tests-job

You Might Be Asking

Why do all my metadata tests show the same "Last computed" timestamp from weeks or months ago?
Are metadata tests supposed to run nightly, and why are they not updating?
The CronJob is firing every night — why are no results being written?
How do I fix metadata test timeouts in a large DataHub environment?
What controls how long the metadata tests job is allowed to run in Kubernetes?

Solution

Confirm the root cause. Verify that the CronJob is launching successfully but being killed before completion by inspecting Kubernetes events for the metadata tests namespace:
```
# Look for SuccessfulCreate followed by DeadlineExceeded on the same job
kubectl get events -n <your-namespace> \
  --field-selector involvedObject.kind=Job \
  | grep -E "metadata-tests|DeadlineExceeded|SuccessfulCreate"
```
A healthy run shows SuccessfulCreate at the scheduled time and a later Complete condition. A failing run shows SuccessfulCreate followed by DeadlineExceeded approximately 12 hours later (or whatever the configured activeDeadlineSeconds is).
Understand the mechanism. The BatchTestEngine scrolls every entity of each targeted type in pages (default batch size: 1 000), sleeping between pages, and submits each page to a fixed-size worker pool. Results are aggregated in memory and written to the store once, after the entire loop completes. Runtime scales with:
```
runtime ≈ (total entities across all targeted entity types)
          × (number of metadata tests)
          / (executor pool size × node CPU allocation)
```
When runtime exceeds the Kubernetes activeDeadlineSeconds, the pod is SIGTERMed and no results are written for that run.
Increase the executor pool size. The default Helm value pins the worker pool to 2, overriding the JVM default of availableProcessors() + 1. Increase this in your Helm values to allow the job to use more CPU in parallel:
```
# values.yaml (or your environment-specific override file)
datahubMetadataTestsJobTemplate:
  config:
    executorPoolSize: 8   # Increase from default of 2; tune to available node CPU
```

Increase CPU requests to match the pool size. If requests.cpu is set too low (e.g., 500m), Kubernetes will not schedule the pod on a node capable of satisfying a larger thread pool. Align the CPU request with the new pool size:

datahubMetadataTestsJobTemplate:
  resources:
    requests:
      cpu: "4"        # Increase proportionally with executorPoolSize
      memory: "4Gi"   # Adjust based on your entity volume
    limits:
      cpu: "8"
      memory: "8Gi"

Extend the job deadline if needed. If runs still approach the deadline after increasing parallelism, raise activeDeadlineSeconds on the CronJob template:
```
datahubMetadataTestsJobTemplate:
  activeDeadlineSeconds: 86400   # 24 hours; default is typically 43200 (12 hours)
```
Reduce the scope of tests where possible. Each additional entity type targeted by a test multiplies the scroll volume. Review your metadata tests and ensure that entity type selectors are as specific as possible. Removing or consolidating tests that target extremely large entity types (e.g., datasets in the millions) will proportionally reduce runtime.

Apply the configuration and verify. After updating Helm values, redeploy and monitor the next nightly run:

# Watch events for the next run
kubectl get events -n <your-namespace> -w \
  | grep -E "metadata-tests"

# Confirm results were written by checking the latest runId
# (adjust index name to your deployment)
curl -X POST "<elasticsearch-endpoint>/<namespace>_test_batchtestruneventaspect_v1/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "query": { "prefix": { "runId": "cron-" } },
    "aggs": {
      "runs": {
        "terms": {
          "field": "runId",
          "size": 10,
          "order": { "_key": "desc" }
        }
      }
    }
  }'

The most recent runId bucket should reflect today's or last night's date. The "Last computed" timestamps in the DataHub UI should update to reflect the completed run.

Additional Notes

All-or-nothing persistence: This is the core behavior that makes the issue appear severe. Because BatchTestEngine writes results only on full completion, a single deadline breach produces zero UI updates — there is no partial result state. A fix to introduce incremental/intermediate writes is tracked in the DataHub open-source project (see Related Documentation). Once that fix is released and deployed, the default configuration will be updated so that large deployments are not affected out of the box.

This is not a fleet-wide regression: Smaller deployments with fewer entities and tests complete well within the default 12-hour window. The issue surfaces only when the product of total targeted entities and test count exceeds what the default 2-thread pool can process in the allotted time.

Incorrect batching behavior: The root cause also includes an incorrect batching implementation that caused excessive writes and hangs under high load, compounding the timeout problem. The long-term fix corrects this batching logic. Until the fixed release is deployed to your environment, the configuration changes above (increased pool size, CPU, and deadline) serve as a reliable mitigation.

DataHub version affected: Confirmed on DataHub v2.1.3. The batching issue traces back to behavior introduced after v1.0.0. Apply the configuration workaround on any version where you observe this symptom; the permanent code fix will ship in a subsequent release.

Entity types not scrolled: Tests only scroll entity types that are explicitly targeted by at least one configured test. Very large indices for entity types not referenced by any test (e.g., dataProcessInstance, query) do not contribute to runtime and do not need to be considered when estimating job duration.