Skip to main content

Fixing Metadata-Test Job Running for Multiple Days on Large Entity Volumes

Angelo Bresci
Updated
```html

Area: Observability Issues

Sub-Area: Metadata Tests Job Performance & Scheduling

Issue

On deployments with large numbers of entities (e.g., 500,000–1,000,000+), the metadata-tests Kubernetes CronJob can take multiple days to complete a single execution. Because concurrent runs are not supported by design, the next scheduled run does not start until the current run finishes. When the concurrencyPolicy is set to Replace, the in-progress job is killed and restarted on each scheduled trigger, meaning the job may never fully complete. Additional contributing factors include an undersized executor thread pool, suboptimal batch parameters, an excessive number of enabled metadata tests, and insufficient resources on dependent backend services (Elasticsearch, GMS, Kafka, PostgreSQL).

Error Messages

  • 120 tasks still pending in actions executor service
  • GMS timeout errors visible in metadata-tests pod logs

You Might Be Asking

  • Should the metadata-test job run on a daily basis?
  • What happens if the next scheduled run starts while the previous one is still running?
  • How can I change the schedule so the job runs less frequently?
  • How do I improve metadata-test job performance without destabilizing my cluster?
  • What does "tasks still pending in actions executor service" mean in the metadata-tests logs?

Solution

Apply the following changes incrementally and monitor the impact on job duration and backend service health after each change.

  1. Fix the concurrency policy first (critical).

    The default Helm value of concurrencyPolicy: Replace kills the running job on every scheduled trigger. For large-entity environments the job will never complete. Change it to Forbid so that a scheduled run is skipped when the previous run is still active.

    # values.yaml (Helm override)
datahubMetadataTestsJobTemplate:
  concurrencyPolicy: "Forbid"

    Keep activeDeadlineSeconds set to a generous ceiling (e.g., 3 days / 259200 s) so that a genuinely stalled job is still reaped automatically.

  2. Adjust the schedule to match your actual completion time.

    If a full run takes 48–72 hours on your entity volume, scheduling daily is counterproductive. Set the cron expression to a cadence that gives the job enough time to finish before the next trigger fires.

    datahubMetadataTestsJobTemplate:
  # Run every 3 days at 05:33 UTC — adjust to your needs
  cron: "33 5 */3 * *"
  3. Tune batch size and delay.

    The default values are conservative. Doubling the batch size and halving the delay reduces round-trip overhead without severely increasing per-batch memory pressure.

    datahubMetadataTestsJobTemplate:
  args:
    - "-u"
    - "EvaluateTests"
    - "-a"
    - "batchSize=2000"
    - "-a"
    - "BATCH_SIZE=2000"
    - "-a"
    - "batchDelayMs=125"

    Warning: Very large batch sizes (e.g., 10 000+) increase the risk of out-of-memory (OOM) errors in the job pod and longer Elasticsearch timeouts. Increase values incrementally and watch pod memory usage. If a batch fails, all entities in that batch are skipped until the next run.

  4. Right-size the executor thread pool.

    The executor pool defaults to a fixed value when EXECUTOR_POOL_SIZE is set explicitly. Set it to N_CPU + 1, where N_CPU is the CPU limit of the job pod. For example, with 8 CPU cores set the pool to 9.

    # Helm values — environment variable override
datahubMetadataTestsJobTemplate:
  env:
    EXECUTOR_POOL_SIZE: "9"   # set to (pod CPU limit) + 1

  # Match or raise CPU resources accordingly
  resources:
    limits:
      cpu: "8"
      memory: "16Gi"
    requests:
      cpu: "4"
      memory: "8Gi"

    Increasing EXECUTOR_POOL_SIZE beyond available CPU cores yields diminishing returns and increases memory pressure from concurrent entity batches held in heap. Scale CPU and pool size together.

  5. Verify and tune backend services.

    The primary bottlenecks are reads from Elasticsearch and writes via GMS (which publishes Metadata Change Proposals to Kafka). The log message "N tasks still pending in actions executor service" indicates that GMS async ingestion calls are queued — typically because GMS or Kafka is saturated.

    • Kafka: Ensure the MCP topic has at least 15 partitions (check the actual broker, not just the Helm value). Verify that Kafka brokers are not CPU- or network-saturated. Do not disable async ingestion on GMS.
    • GMS: Ensure GMS has sufficient replicas and memory. Avoid enabling API-request throttling on GMS while the metadata-tests job is running.
    • Elasticsearch: Increase METADATA_TEST_ELASTIC_TIMEOUT if you raise batch sizes. Monitor Elasticsearch CPU and JVM heap.
    • PostgreSQL: Monitor connection pool saturation if using SQL-backed graph.
    # Example: extend Elasticsearch timeout for larger batches
datahubMetadataTestsJobTemplate:
  env:
    METADATA_TEST_ELASTIC_TIMEOUT: "5m"   # increase from default 1m if needed
  6. Reduce the number of active metadata tests.

    This is often the highest-impact change. Each enabled test is evaluated against every matching entity in every batch. Review all enabled tests and ask:

    • Does this test reflect an active governance or compliance requirement?
    • Does it produce actionable results, or are failures never remediated?
    • Can multiple overlapping tests be consolidated into one?

    Tests that are enabled by default (e.g., "Datasets with Ownership", "Datasets with Domain") may not be needed for every organization. Disabling unnecessary tests directly reduces the per-entity work performed in each batch. Navigate to DataHub UI → Govern → Tests to review, export (as YAML), and disable individual tests.

Additional Notes

  • Concurrency by design: Concurrent metadata-test runs are intentionally not supported. Overlapping executions can produce race conditions when writing test results back to GMS. concurrencyPolicy: Forbid is the correct long-term setting for large environments.
  • Scale ceiling: With appropriate infrastructure (sufficient Kafka partitions, GMS replicas, Elasticsearch capacity), the metadata-test job is designed to scale to millions of entities. Long runtimes in typical deployments are usually caused by one or more of the bottlenecks described above rather than a hard architectural limit.
  • Incremental changes: Change one variable at a time and observe job duration, pod memory, Elasticsearch CPU, and GMS throughput before making the next change. Aggressive simultaneous changes make it difficult to attribute improvements or regressions.
  • OOM risk: Combining a very high EXECUTOR_POOL_SIZE with a very large batchSize loads many entity batches into JVM heap simultaneously. Set -XX:MaxRAMPercentage=75.0 (already a recommended JVM flag) and monitor pod restarts.
  • Verified against Helm chart versions in the acryl-datahub 1.5.x series. Configuration key names may vary in earlier or later chart versions.

Related Documentation

Tags: metadata-tests, performance, scaling, cronjob, concurrency-policy, executor-pool, batch-size, elasticsearch, gms, helm

```