Remote Executor Environment Variables Documentation and Configuration

Users deploying DataHub Remote Executor containers need documentation for environment variables used in the Helm chart configuration. Understanding which variables can be safely modified versus those that should remain at default values is critical for proper deployment support and troubleshooting. Many environment variables are set by default in Helm chart templates, and users need clarity on which variables require documentation and which should not be modified.

You Might Be Asking:

• Where can I find documentation for all Remote Executor environment variables?
• Which environment variables are safe to modify vs. which should remain at defaults?
• What do variables like DATAHUB_EXECUTOR_MODE, DATAHUB_EXECUTOR_INGESTION_MAX_WORKERS, DATAHUB_EXECUTOR_INGESTION_SIGNAL_POLL_INTERVAL, and DATAHUB_EXECUTOR_MONITORS_MAX_WORKERS control?
• Can I remove default environment variables from my values.yaml file?
• Which variables require customer configuration versus being internal-only?

Documentation Location: Official documentation for Remote Executor environment variables is available at: - https://datahubproject.io/docs/managed-datahub/remote-executor/about/

This documentation applies to DataHub version 0.3.10.x and later and covers the environment variables that are intended for customer configuration.

Understanding Variable Categories:

1. Documented Variables (Customer-Configurable):
   • DATAHUB_EXECUTOR_INGESTION_MAX_WORKERS - Controls maximum parallel ingestion tasks
   • DATAHUB_EXECUTOR_MONITORS_MAX_WORKERS - Controls maximum parallel monitor tasks
   • Other variables documented in the official guide
2. Undocumented Variables (Internal/Default):
   • Variables not listed in the official documentation are internal configuration options
   • These are set by Helm chart templates and should not be modified without support guidance

Helm Chart Configuration:

Default variables are defined in the Helm chart templates and values files: - Template file: workload.yaml in the datahub-executor-helm chart - Values file: values.yaml in the datahub-executor-helm chart

Customizing Your Deployment:

# In your custom values.yaml, you can:

# Option 1: Override specific variables (Helm chart 0.0.18+)
env:
  DATAHUB_EXECUTOR_INGESTION_MAX_WORKERS: "10"
  DATAHUB_EXECUTOR_MONITORS_MAX_WORKERS: "5"

# Option 2: Remove variables to use defaults
# Simply omit variables from your values.yaml to use chart defaults
# (Requires Helm chart version 0.0.18 or later)

Best Practices:

1. Use Chart Defaults: If you're using Helm chart version 0.0.18 or later, you can remove default variables from your custom values.yaml if you're not customizing them
2. Only Modify Documented Variables: Only change environment variables that appear in the official documentation
3. Reference Official Docs: Always check the official documentation before modifying any environment variable
4. Version Compatibility: Ensure you're using the correct documentation version for your DataHub deployment (0.3.10.x+)

• Not all environment variables are documented by design - only those intended for customer modification are included in the documentation
• Undocumented variables are internal configuration options that should remain at their default values
• Modifying undocumented variables without support guidance may cause unexpected behavior
• The Helm chart manages default values automatically, so explicit declaration is only needed for customization

• Remote Executor Documentation
• DataHub Executor Helm Chart - workload.yaml
• DataHub Executor Helm Chart - values.yaml

• 3317